search

kent-lee / deviantart-scraper

personal project for downloading artworks from DeviantArt
66 stars 14 forks source link

Resetting file time not permitted on Linux #6

Open chrsmlls333 opened 5 years ago

chrsmlls333 commented 5 years ago

On Ubuntu 18.04, I received this error just after the first user in my config file had been downloaded successfully. This is related to file ownership which is stricter in linux environments. Luckily this only happened when saving to a network or external drive.

I think you should encapsulate this in a try catch and/or implement a config setting to disable this functionality. The ability to sort files is nice but spoofing dates is a little hacky and should not break your script if it fails.

Traceback (most recent call last): File "main.py", line 54, in main() File "main.py", line 40, in main download_users(api, config, args.option) File "main.py", line 11, in download_users result = api.save_users_artworks(config.users, config.save_dir) File "/home/chris/Desktop/deviantart-scraper/lib/deviantart.py", line 204, in save_users_artworks files = self.save_user_artworks(user, dir_path) File "/home/chris/Desktop/deviantart-scraper/lib/deviantart.py", line 183, in save_user_artworks utils.set_files_mtime(combined_files['name'], dir_path) File "/home/chris/Desktop/deviantart-scraper/lib/utils.py", line 42, in set_files_mtime os.utime(file_path, (ts - i, ts - i)) PermissionError: [Errno 1] Operation not permitted

kent-lee commented 5 years ago

@chrsmlls333 I have tested the program on Manjaro 18.04 before with no issue; it appears to me that you have insufficient folder / file permission to make changes to the file attributes. So, one potential solution is to manually set the proper ownership for your download location.

I am not sure why editing file date attributes is considered hacky; it'd be nice if you can let me know the issues regarding this practice, but I do agree with you that I should have an error handler such that the program will end gracefully and not break completely when error happens. Thank you for catching my mistake 👍

I am currently working on another feature of this program, and I will get back on this after the feature is done. If you are curious, Please have a look at my comments in the pull request and here for more details.

chrsmlls333 commented 5 years ago

Yeah that's why I noted that it only happened in certain edge cases of being in the \media folder or others with the permission to write but not modify. Writing to a user directory worked just fine! I don't see any missed files yet other than stories/text files (which I may make a separate issue on), which is really impressive on a challenging and fast-changing site like this.

Sorry to say it was hacky, unconventional would be a better word. I don't think you should change your functionality at all but definitely an error handler, cheers!