Closed julianharty closed 4 months ago
There are essentially two distinct repositories of information:
A reasonable next step is to revise the code that processes the repos hosted on github.com so it uses the progress file provided it exists, otherwise it checks for the TSV file and uses that to populate the progress file. Note: it may be worth exploring what facilities are available to persist the progress file as a Dataframe on disk.
Aa part of the improvements I also plan to rename the current file so it's clear it only processes repos hosted on github.com
utils/initial_data_preparation.py
--> Add script to read TSV file, perform data cleaning, and save results as CSVsrc/github_repo_requests.py
to import the saved preprocessed dataframe performed by utils/initial_data_preparation.py
index=False
when saving the original_github_df.csv to prevent an additional column when it's read in other scripts[github_repo_request_local.py](http://github_repo_request_local.py/)
to be the original_github_df.csv
not the df produced by the script using the github API call
Context
NLnet foundation provides the data using tabs to separate values in a row of data (commonly known as TSV values). Separating values with commas is much more popular (commonly known as csv files). It's easy to mistakenly feed one to a program while expecting the other format.
Let's enhance this project so it can process both formats.
Thoughts
There are various ways this can be done, such as how Microsoft Windows does where the extension part of a filename is used to indicate the format
.tsv
is considered to be tab separated, and.csv
is considered to be comma separated. It's also possible to read and parse the contents of a file and check for commas in each row and tabs in each row then infer the separator based on the results of parsing the contents of the file.