Open epogrebnyak opened 7 years ago
Scrapper template:
Scrapper name:
Job:
Data source:
- URL:
- type of source: [Word, Excel, CSV, HTML, XML, API, other]
- frequency:
Scrapper:
- code:
- control function:
-
Result:
- list of variables:
- endpoint URL:
Testing:
Scrapper | mini-kep |
---|---|
Job | Parse sections of Short-term Economic Indicators (KEP) monthly Rosstat publication |
Source URL | Rosstat KEP page |
Source type | MS Word |
Frequency | Monthly |
When released | Start of month as in schedule |
Scrapper code | https://github.com/epogrebnyak/mini-kep/tree/master/src/kep |
Test health | |
Test coverage | FIXME: So-so |
Documentation | FIXME: So-so |
Controller | TODO: Add here |
CSV endpoint | https://github.com/epogrebnyak/mini-kep/tree/master/data/processed/latest |
List of variables | TODO |
Frontpage data | TODO |
Validation | TODO |
All historic raw data available on internet?
Is scrapper automated (can download required information from internet without manual operations)?
TODO:
DONE:
I guess the Scrapper code URL should be https://github.com/epogrebnyak/mini-kep/tree/master/src/kep
Added JSON dunps of data at https://github.com/epogrebnyak/mini-kep/tree/master/data/processed/json. @Rotzke - can you check if you are comfortable importing this?
pd.to_json
offers several transformations for Dataframe, I chose orient=records
, which seemed most natural, but I'm not sure. For relational database you would have needed rows like:
{"year":1999,"month":1,"EXPORT_GOODS_TOTAL_bln_usd":4.5}
{"year":1999,"month":1,"EXPORT_GOODS_TOTAL_rog":63.7}. etc
at annual, quarterly and monthly frequency.
Is this enough to start discussing common database structure or you would need mroe info?
@Akrishna91 - corrected URL, thanks!
@epogrebnyak Maybe I should tweak the parser to add JSON data directly to DB? Why should we dump it to local files? Except for this everything is fine.
Also, this will be a single collection (relational table) with a separate year in each document (relational row), whole-year statistics and monthly one in the array. Indexed by (year, month), indeed.
@MrBorusLee will create REST API with some endpoints for it after I will have sent him collection structure.
Also, this will be a single collection (relational table) with a separate year in each document (relational row), whole-year statistics and monthly one in the array. Indexed by (year, month), indeed.
Please note there will be data at annual, quarterly, monthly and daily frequency. The final data will be retrieved by frequency, variable names, year, and, where appropriate, month, quarter, and day. Not sure your database structure currently accounts for this. You may also align it more to parser output structure or to end user query structure, which is not formally described yet.
REST API with some endpoints
What part of data pipeline is this? Is this for collection from parser or for end user?
Why should we dump it to local files?
The parser job ends with data/processed
folder as discussed earlier. There will be different parsers and I think it is a mess when they all write to database by themselves. In my view there should be aggregator script that reads parser outputs, verifies them, resolves confilcts with existing data in database, identifies gaps in data and then stores data in database. This way we delimit the responsibility between parser, which must not be aware of database structure and database maintenance scripts.
Instead of JSON parser endpoint you can also read CSV into pandas dataframe by following access_data.get_dfs_from_web(). If you can do this, this eliminates a JSON layer, but you have to think about how you store data from pd.Dataframe to database.
For simple parsers like CBR_USD you can read into database more easily.
@epogrebnyak, answering by quotes:
{
"year":2017,
"data":[
{
"daily":[
{
"day":1,
"vars":[
{
"test":88
}
]
}
]
},
{
"monthly":[
{
"month":1,
"vars":[
{
"test":123
}
]
},
{
"month":2,
"vars":[
{
"test":456
}
]
}
]
},
{
"quarterly":[
{
"quarter":1,
"vars":[
{
"test":789
}
]
},
{
"quarter":2,
"vars":[
{
"test":101
}
]
}
]
},
{
"yearly":{
"vars":[
{
"test":112
}
]
}
}
]
}
Some follow up questions:
"day":1,
is inconvenient for daily data, because is it usually referenced by date csv
module. Why not use pandas own export/import functions at least for some part of this, may be it can save some time on a prototype stage:
For monthly that would be:
year | month | varname | value | date_released | source |
---|---|---|---|---|---|
1999 | 1 | EXPORT_GOODS_TOTAL_bln_usd | 4.5 | 2017-06-26 | mini-kep |
1999 | 1 | EXPORT_GOODS_TOTAL_rog | 63.7 | 2017-06-26 | mini-kep |
date_released
and source
are optional now, but may be required in future to save data vintages.
Tentative list and questions: https://gist.github.com/epogrebnyak/ea407b5ddcee6a26a1967d71f248413c