search

Open-EO / openeo-hub

Source code for openEO Hub, a centralized platform to explore openEO back-end providers.
https://hub.openeo.org
Apache License 2.0
8 stars 3 forks source link

Make MongoDB write more robust against non-alphanumeric keys in backend-supplied JSON #90

Closed m-mohr closed 2 years ago

m-mohr commented 2 years ago

The crawler reports numerous errors when crawling VITO and openEO Platform:

Crawling all backends... (timeout per request is 60 seconds)

  - openEO Platform (well-known document: https://openeo.cloud/.well-known/opene                                                                                                                                                             o)

      - https://openeocloud.vito.be/openeo/1.0.0 ...
          - Downloading https://openeocloud.vito.be/openeo/1.0.0/ ...
          - Downloading https://openeocloud.vito.be/openeo/1.0.0/collections ...
          - Downloading https://openeocloud.vito.be/openeo/1.0.0/processes ...
          - Downloading https://openeocloud.vito.be/openeo/1.0.0/file_formats ..                                                                                                                                                             .
          - Downloading https://openeocloud.vito.be/openeo/1.0.0/udf_runtimes ..                                                                                                                                                             .
          - Downloading details for all 49 collections... (only outputting error                                                                                                                                                             s)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) pref                                                                                                                                                             ixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is                                                                                                                                                              not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
An error occurred while writing to the database (MongoError: The dollar ($) prefixed field '$schema' in 'content.cube:dimensions.x.reference_system.$schema' is not valid for storage.)
m-mohr commented 2 years ago

Mundialis also gives an error:

      - https://openeo.mundialis.de/api/v1.0 ...
An error occurred while gathering collection detail URLs for https://openeo.mundialis.de/api/v1.0 (TypeError: Cannot read properties of undefined (reading 'map'))
christophfriedrich commented 2 years ago

Well, according to the spec that field should be a string or a number, not an object, especially not one with obscure keys like that. I will open an issue at VITO for this.

Nevertheless I agree this shouldn't crash the entire database write for that entry :arrow_right: renamed issue accordingly.

For the Mundialis error I refuse to take any blame when the API response of https://openeo.mundialis.de/api/v1.0/collections is this:


[
  {
    "description": "An internal error occurred while catching mapset from location nc_spm_08!"
  }, 
  400
]

Especially not when that comes with a 200 OK status... IMO just failing and ignoring that response is the best the Hub can do.

m-mohr commented 2 years ago

The API spec is lacking behind. In STAC this is PROJJSON instead of PROJ strings, so what VITO offers is somewhat valid. I simply forgot to update the API spec. See https://github.com/Open-EO/openeo-api/issues/435

For mundialis: Yes, I agree. We should open an issue on their repo instead.

christophfriedrich commented 2 years ago

So apparently the MongoDB server allows storing field names that contain . or $ since version 3.6 (i.e. for a few years now). But when querying such fields there may be problems, which is why the driver implementation prohibits updates that involve these characters in field names. Which doesn't make sense, because we have a perfectly valid use case where we want to store such data, but never query against this very field (it's enough that it comes with the rest when querying against the fields that are used for the index).

Luckily (and at the same time inconsistently), the insert (rather than update) methods of the driver DO allow such inserts. So we can work around this issue by writing our own findOneAndUpdate by combining the three findOne, deleteOne and insertOne. I don't know why those driver engineers are putting us through so much hassle just to achieve something the server accepts anyway... But now it works :)

For the record: I opened that mundialis issue too.

So this issue can be closed.