nlextract / NLExtract

Convert (ETL) and visualize free Dutch geo-datasets.
https://nlextract.nl
GNU General Public License v3.0
155 stars 83 forks source link

Restructure repo dirs/files/execution #249

Open justb4 opened 6 years ago

justb4 commented 6 years ago

Currently each (Stetl-based) ETL process like Top10nl, BRK, BGT etc has its own config/execution mode etc. At the same time all are very similar. Also for a user it is hard to grasp how to perform a specific ETL. This also makes Dockerization harder to develop.

The following needs/can be done to restructure the repo and its (Stetl-based) ETL processes:

Something like

nlextract.sh -p brt/top250 -a brt/top250/options/default.args -a /home/me/nlx/top250.args 

For Stetl an issue has been opened to allow multiple -a args.

Only problem is how to deal with the BAG, which is not Stetl-based and has more extended commandline options. Possibly the default "convert to PostGIS" can be performed by nlextract.sh|py.

justb4 commented 6 years ago

It should be mentioned that this issue is already worked on/merged via PR #244 and #245 by @stvno on a separate restructure repo Branch.

justb4 commented 6 years ago

Stetl (master/latest) ondersteunt nu multiple -a opties. Zie voorbeeld gebruik in top10nl (README): https://github.com/nlextract/NLExtract/tree/master/brt/top10nl/etl . Tevens filenamen gestandaardiseerd, default.args (allowed nu in .gitignore maar niet andere .args bestanden) heeft alle default args, eigen .args hoeft alleen wijzigingen daarop te bevatten bijv alleen DB credentials.