smalers
commented
3 years ago I created a script build-util/harvest-data.bash that processes layers that need to be updated each day. This script could be run on a schedule but for now need to run it manually. Below are some technical issues to resolve:
- The specific workflows to run and resulting output files are handled very specifically. More needs to be done to automate how handled, perhaps by adding annotation comments in the command file to indicate output files that need to be processed.
- The upload process uses ASW s3 sync. Right now the script does not rebuild the distribution and when synced only updates the changed files, which will be the data files. This will work as long as the sync is occurring on one developer's machine. Otherwise, a different developer will have different angular application files and a larger sync will occur. Need to figure out if sync should continue to be used or a simple copy. If copy is used, then periodically it is probably necessary to do a clean of the site, although versioning helps.
Keep this issue open for now as a placeholder to keep making progress on the data harvester.
Some data layers need to be processed each day because they are not available online in a form that can be directly read. For example, the SNODAS data is in old GeoJSON format and needs to be converted to new format. There are existing command files to create the layers. Then need to run the upload script. Need to start with a manual process and then automate so it runs every day.