search

freedomofpress / fingerprint-securedrop

A machine learning data analysis pipeline for analyzing website fingerprinting attacks and defenses.
GNU Affero General Public License v3.0
29 stars 9 forks source link

Crawler cannot read from file and write to database (or vice versa) #55

Closed psivesely closed 7 years ago

psivesely commented 8 years ago

Currently, if one sets use_database=True in fpsd/config.ini under the [crawler] section, then the Crawler will attempt to both read and write from the fpsd (by default) database. This is a problem when one wants to test just the Crawler locally, as we do not initiliaze the hs_history table with any data. It would be nice to be able to read from a local pickle file and write to the database. Just for testing purposes in a local VM. The Sorter should probably always write a pickle file (and we should at least try to use that to keep the one in this repo from getting too stale). This is not super important.

For future reference, @conorsch, I would just run python3 -m pytest test/test_sketchy_sites.py to test the crawler. This would be good for testing the systemd especially because it should complete within a couple minutes, and you can test that it's being restarted.

psivesely commented 7 years ago

@redshiftzero Do you want to give a 1-liner on this one and the development VM test data/ do you think we can close this one?