Some VITO collections' metadata fail to load

[soxofaan]

some VITO collections on https://hub.openeo.org/ fail to load when clicked (e.g. SENTINEL1_GRD), while others work (e.g. TERRASCOPE_S1_GAMMA0_V1)

When checking browser tools I see:

• 500 response on https://hub.openeo.org/api/backends/https%253A%252F%252Fopeneo.vito.be%252Fopeneo%252F1.1/collections/SENTINEL1_GRD -> {"code":"Internal","message":"TypeError: Cannot convert undefined or null to object"}
• 200 response on https://hub.openeo.org/api/backends/https%253A%252F%252Fopeneo.vito.be%252Fopeneo%252F1.1/collections/TERRASCOPE_S1_GAMMA0_V1

[soxofaan]

(@jdries had this problem while demoing openeo collections and STAC)

[christophfriedrich]

Thanks for reporting this. I had a look into it and can now say:

This is happening because for some collections JSON is being returned that contains the key $schema, i.e. a key with a dollar sign. The retrieved data is stored directly into a MongoDB (as document, not string, to enable easy querying) -- and in MongoDB, $ and . are special characters. Therefore the insert fails, so when we try to retrieve the data later there's an undefined where an object was expected.

Crawling gives errors such as:

An error occurred while writing https://openeocloud.vito.be/openeo/1.0.0/collections/S1_GRD_SIGMA0_ASCENDING to the database (Error: key $schema must not start with '$')

I noticed that this is also the case for /udf_runtimes because there the python versions are used as keys, which contain the other problematic character .:

An error occurred while writing https://openeocloud.vito.be/openeo/1.0.0/udf_runtimes to the database (Error: key 3.8 must not contain '.')

That's why the User-defined functions section is missing, too.

The limitation of not being able to store such documents was removed in MongoDB Server v3.6, and I'm pretty sure at least this version is running on the server (@m-mohr). But the drivers officially still don't support it (because there's no syntax to query such documents).

It's the same problem that caused #90 -- I thought I had fixed it, but apparently not.

There seem to be two fixes for this:

• Add the {checkKeys: false} option to the inserts, but sadly the docs don't say what exactly this does (it may switch off more sanitation than wanted)
• Upgrade the driver from v3 to v4, because although nothing about this is mentioned in the changelogs, when I tried it, the problem went away (but this may have other side effects I'd need to investigate)

So far I'm unsure which way to go, but will implement one or the other soon!

[m-mohr]

What are the concerns regarding the driver update? That sounds like the best approach, but I might be missing something...

[christophfriedrich]

You're right, in general it's the much nicer approach -- I hesitated because it wasn't a documented change, just a byproduct: According to the documentation, it should still not work, but well, it does... So here we go. Upgrading also meant I had to check the entire Hub codebase against the breaking changes of this major version jump, but now did it and didn't find any issues. So you can now pull the new state of the dev branch onto the server and give it a go!

[christophfriedrich]

Note 58d6da1, due to the upgrade the minimum Node.js version is now v12 (which shouldn't be a problem since that was released in 2019 and the oldest maintained version is v14).