Extend script support to other source code hosting platforms

commercetest / nlnet

Analysis of the opensource codebases of NLnet sponsored projects.

MIT License

0 stars 0 forks source link

Extend script support to other source code hosting platforms #46

Closed tnzmnjm closed 2 months ago

tnzmnjm commented 3 months ago

So far, the scripts we have are only considering github.com platform. We would like to expand our work to cover other source code hosting platforms like pitchfork.ist, volcanoclient.org, code.wpia.club, etc.

tnzmnjm commented 3 months ago

Add domain-specific DataFrame creation and storage functionality
Implement functionality to read data from a TSV file into a DataFrame.
Perform initial data cleaning to remove null values and duplicate rows.
Extract domains from repourl and create separate DataFrames for each unique domain.
Introduce logic to save DataFrames with more than 10 entries to a specified directory, aiding in domain-specific analysis.
Organise output files in source_code_hosting_platform_dfs directory under the data folder for better management.
Enhance logging throughout the script to improve traceability and debugging capabilities.
ran the github_repo_request_local.py script for codeberg hosting platform and it works.

tnzmnjm commented 2 months ago

Had to fix the git conflict issue for the script utils/initial_data_preparation.py
combine 2 script versions to be able to use the --input-file and --output-folder command-line argument flags and create separate data frames for distinct domains.
Commit is https://github.com/commercetest/nlnet/commit/fa724ff29b7447dba97f6cabc441d6996924aef2

tnzmnjm commented 2 months ago

ran the script with below parameters for codeberg data frame and it works as expected
--clone-dir data/temp_dir/output_files/codeberg_output_files/cloned_repos
--keep-clones
--input-file data/temp_dir/source_code_hosting_platform_dfs/codeberg.org.csv
--output-file data/temp_dir/output_files/codeberg_output_files/updated_local_github_df_test_count.csv
--ttl-file data/temp_dir/output_files/codeberg_output_files/all_data.ttl
--test-file-list data/temp_dir/output_files/codeberg_output_files/test_files_list.txt