Populate databases with examples

[paulmis]

I've populated Firestore and Cloud Storage with output examples. I've made some changes to the schema and bucket layout:

• renamed runs to scans
• added pipeline tags (scan, auxiliary)
  • scan pipelines produce final output to display in the app
  • auxiliary pipelines aid in the data collection process, and their output should largely be ignored by the frontend
• moved indices and insights from the pipeline to scans:
  • pipelines dump their pipeline data object (PDO, see here) to the pipeline collection
  • when a scan is finished, the pipeline dumps the frontend-relevant data into the scan collection
• grouped scans by fields rather than users

Please see if you agree with all these changes!!

The example data is here:

• Firestore (scan c6bd6bf3-8ea9-40f3-ac73-cd46521a5270)
• Cloud Storage

One important thing is that since we're supporting cameras that potentially don't capture all 3 RGB channels, the RGB index outputs the "default" image. However, that module might not be in a given pipeline, so the frontend needs to account for that.

Aaand regarding image sizes, they can grow very large - the example full-size RGB image is 0.5GB. We will downsize and compress the images on the backend to make it easier for you to display thumbnails and avoid progressive loading for the moment. What sizes (resolution, weight) do you guys want us to produce @nadinekuo @RD211?

• [x] Set real boundary coordinates for the example data field
• [x] Add a Mat::fread overload for purely grayscale loading

[nadinekuo]

By "the RGB index outputs the "default" image", do you mean the plain image? In case the pipeline does not contain the "RGB" module, would we just rely on the satellite images provided by Google Maps? Also, is the NDVI .tif image supposed to be in grayscale?

[paulmis]

@nadinekuo

As for image size and weight, I'd say around 1024 x 1024 px (around 5 MB) to have decent quality but keep the app a bit responsive ;)

The downscaled image is 5000x4000 and is 28.5MB, which comes out at around 1.5MB/MP. Do you think this is good, or should it be a bit bigger?

By "the RGB index outputs the "default" image", do you mean the plain image?

Yeah, as in the normal color image.

In case the pipeline does not contain the "RGB" module, would we just rely on the satellite images provided by Google Maps?

Yes

Also, is the NDVI .tif image supposed to be in grayscale?

Yeah, as far as I can tell that's the format to store grayscale. What you can do is color it using an arbitrary color scheme (e.g. for NDVI is RdYlGn to transform it into a color scale. Can you make an issue for that for the frontend?

[nadinekuo]

Also, the bucket layout is as follows: terraform-scans/{field}/{MM-YYYY}/{scan}/pipelines/{pipeline} whereas {field} is {user} at the moment?

[nadinekuo]

The downscaled image is 5000x4000 and is 28.5MB, which comes out at around 1.5MB/MP. Do you think this is good, or should it be a bit bigger?

I'm afraid that even the downscaled image is too large (users typically don't even have close to that many pixels), but we can resize on the frontend after fetching them

[paulmis]

Also, the bucket layout is as follows: terraform-scans/{field}/{MM-YYYY}/{scan}/pipelines/{pipeline} whereas {field} is {user} at the moment?

Might have forgotten to change it, please feel free to switch it to the field id.

I'm afraid that even the downscaled image is too large (users typically don't even have close to that many pixels), but we can resize on the frontend after fetching them

No, I meant I will downsize it to the target size/resolution at the backend, but do you want me to do it according to the 5MB size or 1MP resolution?

[nadinekuo]

No, I meant I will downsize it to the target size/resolution at the backend, but do you want me to do it according to the 5MB size or 1MP resolution?

I'd say around 5MB, as 28.5 MB is most likely gonna lead to a drastic slowdown 😅 The ideal setting would be having different resolutions per zoom level but that is hard