bellingcat / auto-archiver

Automatically archive links to videos, images, and social media content from Google Sheets (and more).
https://pypi.org/project/auto-archiver/
MIT License
578 stars 60 forks source link

breaking changes: refactor configs + fixes #33

Closed msramalho closed 2 years ago

msramalho commented 2 years ago

Breaking changes.

djhmateer commented 2 years ago

Thanks - my PR is coming today!

djhmateer commented 2 years ago

Wow - some great work in here! I can't immediately see why the google drive code errored. It does look like it saved some files up there successfully. Have emailed you directly about a problem with tiktok-downloader repo. Once that is running I may be able to get this working and test out the GD implementation problems.

Are you going to merge this into dev and we can work from there?

msramalho commented 2 years ago

The bug @loganwilliams found with the gd storage + twitter archiver was related to the generate_media_page which was calling get_cdn_url before the file was uploaded, and GDStorage, unlinke S3, will not be able to get the URL before uploading. fixed by removing this line: 6499161 (#33) (the line was duplicated below, where it could actually be called)

msramalho commented 2 years ago

All bugs and change requests have been fixed and applied, thanks a lot for the useful feedback, the example.config.yaml is now clearer