search

perslev / CanvasSync

Synchronise modules, assignments and files located on a Canvas by Instructure web server to your local computer.
MIT License
68 stars 33 forks source link

String indicies must be integers #11

Closed SShah7433 closed 5 years ago

SShah7433 commented 6 years ago

When running CanvasSync, some users get the following error:

Traceback (most recent call last):
  File "/usr/local/bin/canvas.py", line 177, in <module>
    run_canvas_sync()
  File "/usr/local/bin/canvas.py", line 120, in run_canvas_sync
    main_menu(settings)
  File "/usr/local/bin/canvas.py", line 135, in main_menu
    main_menu(settings)
  File "/usr/local/bin/canvas.py", line 142, in main_menu
    do_sync(settings, "")
  File "/usr/local/bin/canvas.py", line 158, in do_sync
    synchronizer.sync()
  File "/usr/local/lib/python3.7/site-packages/CanvasSync/CanvasEntities/synchronizer.py", line 123, in sync
    course.sync()
  File "/usr/local/lib/python3.7/site-packages/CanvasSync/CanvasEntities/course.py", line 169, in sync
    child.sync()
  File "/usr/local/lib/python3.7/site-packages/CanvasSync/CanvasEntities/folder.py", line 142, in sync
    self.add_files()
  File "/usr/local/lib/python3.7/site-packages/CanvasSync/CanvasEntities/folder.py", line 90, in add_files
    if file[u"id"] in self.black_list:
TypeError: string indices must be integers

This is caused by CanvasSync/CanvasSync/CanvasEntities/folder.py line 86 (files = self.api.get_files_in_folder(self.id)) and/or line 98 (folders = self.api.get_folders_in_folder(self.id)) returning 'unauthorized' as the status.

Mibo5354 commented 5 years ago

Is this issue ever going to be fixed?

SShah7433 commented 5 years ago

I tweaked the code, but ended up writing my own canvas export code. It's super simple, doesn't have a bunch of features, but it works.

It also is not limited by the number of modules in a course.

Gist of Canvas Download Script

Mibo5354 commented 5 years ago

I get errors with your script. Windows

NotADirectoryError: [WinError 267] The directory name is invalid: 'D:\\Canvas\\BIOLOGY I: Fundamentals of Biomedical Science - Combined Sem 1&2 2019'

On linux

Traceback (most recent call last):
  File "dump_canvas.py", line 65, in <module>
    module_dir = os.path.join(os.path.decode('utf-8').join(output_dir, course_name), module_name)
AttributeError: 'module' object has no attribute 'decode'

Edit: Nvm about linux, I found out it was using python 2 and I need to turn the output_dir into a raw string because of "\Users" messing with unicode. Edit2: now it runs but stops midway.

Traceback (most recent call last):
  File "dump_canvas.py", line 62, in <module>
    file_content = session.get(file_info['url'])
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 506, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 449, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python3/dist-packages/requests/models.py", line 305, in prepare
    self.prepare_url(url, params)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 379, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://?
SShah7433 commented 5 years ago

The error on windows is most likely due to the ampersand and colon in the file path (directory name).

I ran my code on a mac using python3.7. The traceback shows line 65 including a decode method on a module, which I do not see in my code?

Regardless, I updated my code to only include alphanumerics in the file name, which should fix the issue you had on windows.

Let me know if it works.

SShah7433 commented 5 years ago

The error you showed in your edit maybe something specific to your course. I would need to see the URL for the file.

Could you either run a debugger or add a print statement before line 62 with print(file_info['url'])?

Mibo5354 commented 5 years ago

Running your updated script on windows

https://myuni.adelaide.edu.au/files/4188861/download?download_frd=1&verifier=1JMjn0RdS2smkryJ9nT6YW9drJrzwogdqzCNUUx0
                Processing File: ORAL CASE PRESENTATION EXAMPLE.docx
        Processing Module: Week 1
        Processing Module: Week 2 (Quiz due **)
        Processing Module: Week 3 (Quiz due **)
https://myuni.adelaide.edu.au/files/4011816/download?download_frd=1&verifier=SRPEgbZJhH2PYORxloEhJqxZwrm4lyEdYcGvdwcb
                Processing File: Vital Signs Learning Outcomes.docx
Traceback (most recent call last):
  File "dump_canvas1.py", line 77, in <module>
    with open(file_name, 'wb') as o_file:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\ryanl\\OneDrive\\Documents\\Documents\\2019\\Canvas\\Clinical Practice I Part 1\\Week 3 Quiz due \\Vital Signs Learning Outcomes docx'
Mibo5354 commented 5 years ago

Running updated script on linux

https://myuni.adelaide.edu.au/files/4337272/download?download_frd=1&verifier=iMILVzWLUgylD6Er5KXlFFYcZquphBhgfc0POimX
                Processing File: Y1 2019 Practice Case S1 Large Group - Student version.pdf
        Processing Module: Case 1 - Lola Smith - Chest Pain
https://myuni.adelaide.edu.au/files/3863762/download?download_frd=1&verifier=BRKI1fMlJEN09kC5EWguDul5jROXvAyF6nWnKT62
                Processing File: Yr 1 2019 Smith L - Pre-Reading.pdf
https://myuni.adelaide.edu.au/files/3863767/download?download_frd=1&verifier=GMS4ztyrbxvZ4LZMSlv1TF6enGWPfFjxGW6A5xmG
                Processing File: Yr 1 2019 Smith L - Student Agenda - Session 1 .pdf
https://myuni.adelaide.edu.au/files/3863763/download?download_frd=1&verifier=VSKAY1JnCOWpWrCAwCnAzD6dSIOfYJsX0GHbbiPG
                Processing File: Yr 1 2019 Smith L - Session 1.pdf
https://myuni.adelaide.edu.au/files/4319050/download?download_frd=1&verifier=5r7Bc7c2CCWTGdgYglEGtBXuARfJ9OSYnc36JEfM
                Processing File: Yr 1 2019 Smith L - Case Presentation Session 1.m4a
https://myuni.adelaide.edu.au/files/3863768/download?download_frd=1&verifier=18tZzIR2g9T9VdyUlCYrvzvksIxwKu0tGAD5kq60
                Processing File: Yr 1 2019 Smith L - Student Agenda - Session 2.pdf
https://myuni.adelaide.edu.au/files/3863764/download?download_frd=1&verifier=D1L8HSHAFCAnEp3ORpknhGBJbXuFjIUGxIntx3m9
                Processing File: Yr 1 2019 Smith L - Session 2.pdf
https://myuni.adelaide.edu.au/files/4319051/download?download_frd=1&verifier=CO74nUbCuSW0FveJbvqCLgTrRAv6CPTwIszVmdS4
                Processing File: Yr 1 2019 Smith L - Case Presentation Session 2.m4a
https://myuni.adelaide.edu.au/files/3863769/download?download_frd=1&verifier=oFdy8cuVxStA0ICNadKQQ7D5r3VzmwkCP6vzYob3
                Processing File: Yr 1 2019 Smith L - Student Agenda - Session 3.pdf
https://myuni.adelaide.edu.au/files/3863765/download?download_frd=1&verifier=AHVFggfloCtV3eN1IhTCcgIqvC7nhCH2B8PWpb3i
                Processing File: Yr 1 2019 Smith L - Session 3.pdf
https://myuni.adelaide.edu.au/files/4319052/download?download_frd=1&verifier=JTU5plTGUXOxXubJksiFDKzl733AduNXm5zPwWJc
                Processing File: Yr 1 2019 Smith L - Case Presentation Session 3.m4a
https://myuni.adelaide.edu.au/files/3863770/download?download_frd=1&verifier=hgO4m8CQAVxh3bdmYmG8ossIJN3evuccjb0i4VZT
                Processing File: Yr 1 2019 Smith L - Student Agenda - Session 4 .pdf
https://myuni.adelaide.edu.au/files/3863766/download?download_frd=1&verifier=v9BKpwxzTATUFTkaUsVcObzZjDA6uv5OrBZJFpsU
                Processing File: Yr 1 2019 Smith L - Session 4.pdf
https://myuni.adelaide.edu.au/files/3863759/download?download_frd=1&verifier=5avKy07Y7KQimGtpub6XRDQKZXTQToWuaCOJBXbd
                Processing File: Yr 1 2019 Smith L - Case objectives.pdf
        Processing Module: Case 2 - David Marsden - Chest Pain & SOB

                Processing File: Yr 1 2019 Marsden D - Pre-Reading.pdf
Traceback (most recent call last):
  File "dump_canvas_nix.py", line 67, in <module>
    file_content = session.get(file_info['url'])
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 506, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 449, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python3/dist-packages/requests/models.py", line 305, in prepare
    self.prepare_url(url, params)
  File "/usr/lib/python3/dist-packages/requests/models.py", line 379, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://?
Mibo5354 commented 5 years ago

For linux, it's probably because some files are still locked therefore no url provided.

SShah7433 commented 5 years ago

Lets stick with solving the Linux bug. (Easier to solve one problem at a time).

I took out the file path cleanup code. The file not available makes sense, and is easy to solve.

Could you give it another try?

Mibo5354 commented 5 years ago

It works now, thank you.

vanalmsick commented 5 years ago

TypeError: string indices must be integers Real Error: Unauthorized: User is not authorized for this action.

I had the same error. I am not so sure that it is really a bug since 'unauthorized' means that the user is not allowed/authorized to upload/modify files on Canvas (has just read rights). Therefore, the add_files function returns an error. Maybe an error-handler should be added that returns a more precise error code like: 'User not authorized to add/modify files and folders in ...'

perslev commented 5 years ago

Hi, thank you for your feedback.

This originally reported issue should now be resolved in the latest version (v0.2.4). The solution is, however, a quick fix as CanvasSync still does not systematically handle responses to unauthorized requests.

A general error handling mechanism should be introduced.