CartoDB / raster-loader

https://raster-loader.readthedocs.io
Other
15 stars 4 forks source link

Modularization refactor #9

Closed tcmetzger closed 1 year ago

tcmetzger commented 2 years ago

To future-proof this package and make it easier to add additional functionalities, we need to modularize the code.

We propose three functional areas:

brendancol commented 2 years ago

@giancastro @tcmetzger Hey I have a PR up here: https://github.com/CartoDB/raster-loader/pull/23/files

tcmetzger commented 1 year ago

@brendancol Chunking needs to remain a user decision, because there is an upload limit in BigQuery that needs to be respected.

tcmetzger commented 1 year ago

@brendancol the BigQuery module should include a simple integration test (without the complex BigQuery SQL), i.e. a bytestring comparison as non regression test. (Integration test is its own task, see Issue #18)

tcmetzger commented 1 year ago

When generating the bytestring record (currently in .tobytes()), we need to use endian-independent code. See https://github.com/CartoDB/raster-loader/issues/29 for details!