WIP: screenshots and hashing

bellingcat / auto-archiver

Automatically archive links to videos, images, and social media content from Google Sheets (and more).

https://pypi.org/project/auto-archiver/

MIT License

578 stars 60 forks source link

WIP: screenshots and hashing #16

Closed loganwilliams closed 2 years ago

loganwilliams commented 2 years ago

[x] Support adding filehash to google sheet
[x] Support adding page screenshot to google sheet
[ ] The new methods for getting file integrity hash and a screenshot are called in the exact same way in each archiver. This should be refactored as some kind of middleware that could be applied to an arbitrary archiver. (Or something like that.)
[ ] Support detecting multiple names for columns (or over-riding defaults via a command line attribute -- this became an issue because the Ukraine datasheet was pre-existing and has different naming conventions. In this branch I have just changed the defaults to match, but this isn't a great solution because the name in the Ukraine sheet is a poor default.)

msramalho commented 2 years ago

At this stage, the changes look good although the youtube-dl timestamps might still be an issue. I can help with the last checkitem: Support detecting multiple names for columns

loganwilliams commented 2 years ago

Thanks! Yeah, I'd like to make the times consistent in UTC before merging into main. I think we need to test:

a few different providers (Twitter, Youtube, Facebook, others?)
a few different computers/servers/configurations (YoutubeDL seems to be giving me twitter timestamps in CET, even over a UK or US VPN. I assume that it's based on some region detection that Twitter is doing... but I'm not sure.)

Once we've understood this behavior we should make sure that dates from the auto_archiver are consistently UTC.

Another task: currently the screenshot generator isn't a headless browser, which could cause problems with running this on a server. Should be a simple change, maybe to Chromium instead of Firefox/Gecko. Or maybe you can figure out how to get headless mode working on Firefox.

loganwilliams commented 2 years ago

Making it headless turned out to be fairly straightforward, but it required making sure the Firefox and geckodriver version numbers are compatible

loganwilliams commented 2 years ago

Timestamps are fixed :)

loganwilliams commented 2 years ago

@msramalho I tried to define the remaining two to-do items more precisely. I think those are the next steps, then its ready for your review or any other suggested changes you have.

msramalho commented 2 years ago

Had a look and these changes are great. I will work on the 2 checkbox items TODO from your original comment in another branch