weiji14
commented
11 months ago Can we move this to https://github.com/Clay-foundation/model/discussions/categories/ideas? It would be good to keep Issues focused on tasks that contributors can help with, and Discussions for ideas like these that might be a longer conversation.
This only to document a deviation from earlier plans, and the benefits that it provides.
Originally (this is the 1st attempt from August'23 ), we had planned to create image embeddings with absolute anchors. The idea being that in such way we would create a semantic db of Earth. It also leaned on the zxy scheme and its nested nature, and tooling, to semantically nest local-global semantics.
As we started to build it, we realized that this simplistic view inherits more challenges than the current approach. E.g. zxy corresponds to different resolutions depending on latitude, there is no wrap around of Earth, we don't really know how to handle time, or instruments, ...
The current approach, instead of creating a set of intrinsic absolute embeddings of Earth, creates a model that then one can run inference for other images. E.g. if you run inference on USA images in 2022, it would create such embeddings for such location and time. Likewise trying with other resolutions, instruments, ... of course, the more different the input data at inference time from the training data, the more error we can expect.
However, this approach of making a model, then spatiotemporal embeddings through inference has a set of extra benefits, like creating them with less open or non-open imagery, after events that significantly change the semantics of the location (e.g. fires, floods, war, ...), for any sets of times, specific RoIs, incremental updates, or even on-the-fly.
This issue is to ensure we understand the tradeoff, no actions requested other than correction and clarifications.