Open carlosug opened 1 month ago
visualisation is at readmes_statistics.py
- it does:
basic_statistics.py
has more detailed info:
``,
) and their counts per line.updates:
get_Links.py
-> output.txt -> run extractor_all.py
-> run extractor_install.py
-> output installation_features_all.json
updates:
basic_statistics.py
performs descriptives on many subjects such ids with and without installation instructions as well as visuals about programming languages been used per year (issue on data_created
versus data_updated
)code_blocks_stats.py
- the issue is that I cant create multiple heatmap per year - it only plots barsvisualisation is at
readmes_statistics.py
- it does:* Keyword Extraction: For each result in an item, the script preprocesses the text, uses CountVectorizer to find the top 10 keywords, and updates the keyword counters. * Heatmap Preparation: The script aggregates keywords for each repository and creates a DataFrame of the top 10 keywords. * Plot Heatmaps: Heatmaps are created for each repository with the top 10 keywords and their counts.
readme_statistics.py
removed as it does not provide any value.
developed the script somef ->
statistics.py
. it does:* Preprocessing Functions: Functions to preprocess text and check for external links. * Initialize Statistics: Initializes dictionaries to count installation methods, results, code blocks, images, hyperlinks, and tokens for each repository. * Process Data: Iterates through each key in the JSON data to populate the dictionaries. * Calculate Installation Methods: Calculates the number of installation methods per repository. * Keyword Analysis: Analyzes the occurrence of specific installation keywords. * Trends and Observations: Prints trends based on the keyword analysis. * Installation Types: Categorizes installation types based on keywords. * Compile Statistics: Compiles overall statistics into a DataFrame and prints it.
statistics.py
to be removed
main updated:
hyper_links_stats.py
to analyse the repos per yearcode_block_stats.py
basic_statistics.py
remains the MAINinstallation_readmes.json
into division of senteces in structured.py
main updated:
extractor_all.py
and got 716 repos out of 1800installation_readmes-01.json
04-basic_statistics.py
runs over all again and 05-hypter_links.stats.py
and 06-code-block-stats.py
- double check how to process unknown
data in the script
developed the script somef ->
statistics.py
. it does: