calculate and visualise statistics for the dataset

[carlosug]

developed the script somef -> statistics.py . it does:

• Preprocessing Functions: Functions to preprocess text and check for external links.
• Initialize Statistics: Initializes dictionaries to count installation methods, results, code blocks, images, hyperlinks, and tokens for each repository.
• Process Data: Iterates through each key in the JSON data to populate the dictionaries.
• Calculate Installation Methods: Calculates the number of installation methods per repository.
• Keyword Analysis: Analyzes the occurrence of specific installation keywords.
• Trends and Observations: Prints trends based on the keyword analysis.
• Installation Types: Categorizes installation types based on keywords.
• Compile Statistics: Compiles overall statistics into a DataFrame and prints it.

[carlosug]

visualisation is at readmes_statistics.py - it does:

• Keyword Extraction: For each result in an item, the script preprocesses the text, uses CountVectorizer to find the top 10 keywords, and updates the keyword counters.
• Heatmap Preparation: The script aggregates keywords for each repository and creates a DataFrame of the top 10 keywords.
• Plot Heatmaps: Heatmaps are created for each repository with the top 10 keywords and their counts.

[carlosug]

basic_statistics.py has more detailed info:

• Line Counts: Calculate the total number of lines, blank lines, unique lines, and the ratio of blank and repeated lines.
• Token Counts: Calculate the number of tokens, characters, repeated token ratio, unique tokens per line, and average tokens per line.
• Code Block Counts: Count the occurrences of code block markers (``,) and their counts per line.
• Token Length: Calculate the average length of tokens.
• N-gram Ratios: Calculate the ratio of unique bigrams and trigrams.
• Stop Words: Count the number and ratio of stop words, and stop words per line.
• Uncommon Words: Count the number of uncommon words (words not in WordNet).

[carlosug]

updates:

• run the cli somef but stop at N = 256. TODO: rerun and get all if needed now
• dataset created by get_Links.py -> output.txt -> run extractor_all.py -> run extractor_install.py -> output installation_features_all.json

[carlosug]

updates:

• the run cli somef still crashes due to storage limitation. only N=300 readmes has been parsed
• now the script basic_statistics.py performs descriptives on many subjects such ids with and without installation instructions as well as visuals about programming languages been used per year (issue on data_created versus data_updated)
• more detailed report on code blocks is stored in code_blocks_stats.py - the issue is that I cant create multiple heatmap per year - it only plots bars

[carlosug]

visualisation is at readmes_statistics.py - it does:

* Keyword Extraction: For each result in an item, the script preprocesses the text, uses CountVectorizer to find the top 10 keywords, and updates the keyword counters.

* Heatmap Preparation: The script aggregates keywords for each repository and creates a DataFrame of the top 10 keywords.

* Plot Heatmaps: Heatmaps are created for each repository with the top 10 keywords and their counts.

readme_statistics.py removed as it does not provide any value.

[carlosug]

developed the script somef -> statistics.py . it does:

* Preprocessing Functions: Functions to preprocess text and check for external links.

* Initialize Statistics: Initializes dictionaries to count installation methods, results, code blocks, images, hyperlinks, and tokens for each repository.

* Process Data: Iterates through each key in the JSON data to populate the dictionaries.

* Calculate Installation Methods: Calculates the number of installation methods per repository.

* Keyword Analysis: Analyzes the occurrence of specific installation keywords.

* Trends and Observations: Prints trends based on the keyword analysis.

* Installation Types: Categorizes installation types based on keywords.

* Compile Statistics: Compiles overall statistics into a DataFrame and prints it.

statistics.py to be removed

[carlosug]

main updated:

• generate script for hyperlink hyper_links_stats.py to analyse the repos per year
• generate block code analysis per year in code_block_stats.py
• basic_statistics.py remains the MAIN
• split the json file installation_readmes.json into division of senteces in structured.py

[carlosug]

main updated:

• re run extractor_all.py and got 716 repos out of 1800
• last file: installation_readmes-01.json
• 04-basic_statistics.py runs over all again and 05-hypter_links.stats.py and 06-code-block-stats.py - double check how to process unknown data in the script