Refactoring Code: Possible Structure

evaristoc commented 7 years ago

Steps //MAKES POINT PER TABLE, PREPARE A DATA FLOW//

[ ] Social Data Extraction: INPUT: different sources; OUTPUT: different raw data files
[ ] Platformtable Database pre: INPUT: different raw data files, affecting only those links that are included in rules; OUTPUT: a dataset that resembles the database in different aspects, keeping some areas empty for filling in
[ ] Crawled Platformtable Database pre: INPUT: a database-like dataset; OUTPUT: a database-like dataset with crawled data
[ ] Platformtable Dataset for classification: INPUT: a database-like dataset with crawled data required for the classification procedure; OUTPUT: a first dataset for classification
[ ] Crawled Platformtable Database pre with Classes: INPUT: a first dataset for classification + operational definition of classes dataset; OUTPUT: collating a full dataset with ALL the links classified, currently called botdata + Platformtable Database pre filled, including regex forms
[ ] Crawled Platformtable Database pre with Classes and Link Comparisons: INPUT: the dataset currently known as botdata + Crawled Platformtable Database pre with Classes; OUTPUT: collating data calculated based on what was found in botdata
[ ] Curriculum Data Extraction: INPUT: freecodecamp website or similar; OUTPUT: a dictionary per section with some data transforms + subjects, usually called cv
[ ] Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added INPUT: CV + BOW + Crawled Platformtable Database pre with Classes and Link Comparisons; OUTPUT: collating of data into the dataset pre filled with metrics of subject relevance
[ ] ETL of Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added: verify compliance before loading, OBS: also includes other tables!!
[ ] Database file: platformstable: INPUT: ETL of Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added; OUTPUT: updates in the firebase database
[ ] Database file: fcc_subjects: INPUT: ETL of Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added; OUTPUT: extraction of unique fcc_subjects as captured from the whole list (regex ones)
[ ] Database file: cv: INPUT: CV dict file; OUTPUT: cv dict table

evaristoc commented 7 years ago

Options for dataflow:

evaristoc commented 7 years ago

Trying with www.draw.io.

See resulting xml at: https://github.com/evaristoc/fCC_R3_DataAnalysis/tree/development/docs (Untitled Diagram.xml)

evaristoc commented 7 years ago

Possible order:

Social Data Extraction, Curriculum Data Extraction => src/0_dataextraction
Platformtable Database pre => src/1_plfdatabasepre
Crawled Platformtable Database pre => src/2_crawler
Platformtable Dataset for classification, Crawled Platformtable Database pre with Classes, Crawled Platformtable Database pre with Classes and Link Comparisons => src/3_classifierANDcomparison
Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added => 4_plfdatabaseplus
ETL of Crawled Platformtable Database pre with Classes and Link Comparisons with Subject Metrics added, Database file: platformstable, Database file: fcc_subjects, Database file: cv => src/5_databasefiles

evaristoc commented 7 years ago

TASKS:

[ ] Define parameters for each script so some more flexibility in use is reached as long as it is specific to the control area the script belongs to
[ ] Get rid of global variables; all the project stateless as possible for now
[ ] Define a main to call all the scripts
[ ] Doc the scripts as much as possible, or at least a README indicating what it is about

evaristoc / fCC_R3_DataAnalysis

Refactoring Code: Possible Structure #23