GHA

[smnorris]

Try and run the full job using GHA runners.

Requires:

• location to serve source and output files (object store)

• reworking scripts to minimize data held in the db - ideas:

  • download data from WFS to object store rather than loading to db
  • process one 250k tile at once

[smnorris]

Seems it is possible to generate the matrix to process the tiles in parallel, but I cancelled the proof of concept test run because it was not progressing... is it reasonable to make 89 concurrent requests to a single parquet file on object storage? Need to:

• test creating column with actual tile values of interest, a LIKE query does not seem to be pushed down to arrow
• confirm the parquet file is sorted by tile (a single request is slower than expected)
• if many concurrent requests are still an issue, downloading the entire dataset and loading required tile from local file works fine too, the job is still well below gha size limits

[smnorris]

Still sluggish to get going but seems to work now that the files are properly formatted: https://github.com/bcgov/CE_integratedroads/actions/runs/8620376591/job/23627218010

[smnorris]

Potential improvements:

• splitting up the single ce-integratedroads workflow into reusable workflows would be valuable - download; preprocess; final process. Mostly just for development/testing - there is no need to download/preprocess more than once unless something has changed
• the output tiled parquet files can likely be better organized (folder/label)