search

ausocean / openfish

OpenFish is an open-source system written in GoLang for classifying marine species. Tasks involve importing video or image data, classifying and annotating data (both manually and automatically), searching, and more. It is expected that OpenFish will use utilize computer vision and machine learning techniques.
https://ausocean.github.io/openfish/
Other
6 stars 0 forks source link

Google cloud datastore does not support inequality filters on multiple properties. #23

Open scott97 opened 1 year ago

scott97 commented 1 year ago

When checking if a videostream is within the specified timespan, ideally we would filter out all videostreams that ended before the specified range, and started after the specified range. This involves making a datastore query with two inequality filters on two different properties (our start time and end time). This is forbidden, so we need a work around.

One option is to filter out all videostreams that ended before the specified range with our query, and then on the application side we can filter out all videostreams that started after the specified range.

Current error message

HTTP/1.1 500 Internal Server Error
Date: Wed, 28 Jun 2023 06:46:18 GMT
Content-Type: application/json
Content-Length: 161
Connection: close

{
  "message": "could not read from datastore: rpc error: code = InvalidArgument desc = Cannot have inequality filters on multiple properties: [EndTime, StartTime]"
}

Relevant documentation https://cloud.google.com/datastore/docs/concepts/queries#restrictions_on_queries

scruzin commented 11 months ago

This is a fundamental restriction of the datastore and unlikely to change anytime soon.

Time queries are of course fine providing the inequality applies to one property, e.g., Timestamp > X and Timestamp < Y.

If we find ourselves needing to do time-based queries AND space-based queries, we should consider adopting a "space-time GeoHash".