Web crawler from TickBase project, summer 2021. The project aimed to:
Code from this repository may only be copied or re-used with the permission of Garrett Wells or Lucas Sheneman.
Below is a summary of subdirectories and their contents.
Contains source Python 3 code. This code is a combination of APIs for interfacing with various web sources and crawling the web. It also contains the source code for tickpicker.py: a command line application which provides the quick access point to executing queries and collecting data. Use it to:
After cloning this repository, make sure there is a folder named _data_dump in the same parent directory as the code_ folder. Output from the command line app will be placed in this folder.
Likewise, if you intend to run an automated search from a CSV of keywords make sure you have a folder named searches in the top level directory.
This should be done by default... but you will receive error messages if not.
# after cloning repository...
# and installing python3...
# and any other necessary libraries as prompted...
$ python3 tickpicker.py
This folder is the default destination of the CSV files containing metadata from successful searches. Output files follow the convention:
dataRepositoryName_dataType_searchKeyword_searchDate.csv
This folder contains some example output from searches on various databases. Each output file name contains the source, search keyword, and date of search. Warning this folder may not always be populated as it does not contain functional code is not part of any dependency tree.
Contains example CSV files full of search keywords (mostly pertaining to ticks :)). Use these for testing or examples of how to construct your own search files. Selecting option 2 from the main menu in tickpicker.py
to select a CSV to run queries with.