Cronjob ftp script - Githubissues

mapic / sdk

Scripts in Bash and NodeJS for interacting with Mapic Services

https://mapic.io

GNU Affero General Public License v3.0

0 stars 1 forks source link

Cronjob ftp script #1

Open knutole opened 8 years ago

knutole commented 8 years ago

Script for pulling data from FTP server

[x] Script for connecting to ftp
[ ] rename config.json to access.ignore.json
[ ] put ftp config in dataset-config file
[ ] move all config files (incl. datasets) to config folder
[ ] Pluggable further actions, like replace-datasets
[ ] Registering cronjob with Engine API
[ ] Figure out how best to do cronjobs
[x] Check for those datasets that are changed compared to existing datasets (in cube or singular)
[x] refresh stored query after updating cube
[ ] fix bug with "last date" in animator
[ ] fix query bug (last date)
[x] automatically do query on import and when updating cube
[x] clean up!
[x] update code on other portals
[ ] create an ftp project from scratch (ie. without pre-existing project, cube, layer)
[ ] clean up folder
[ ] REDACT ftp ..

knutole commented 8 years ago

What is better, to have an FTP folder that is automatically imported and periodically checked and updated? Or to keep existing structure of importing, and simply finding new files in FTP folder?

They intersect somewhat: having a cube id + ftp folder, the script in any case needs to check for latest files...

knutole commented 7 years ago

Updated Wu.CubeLayer after updating cube...

cube is a dataset, upon many layers can be created. When changing cube, all corresponding layers should be updated with latest cube... (Better solution is ofc to store cube_id only on Layer, and request each time... More timeconsuming, but cleaner.)

Current solution: Create endpoint @ engine for updating cube, which will search through all Wu.Layers that has cube and replace this cube string.

Simplest solution: add layer_id to ftp script dataset config, and update this layer with new cube.