Discussion: How to merge-in or incrementally add data?

[jlaura]

I am not 100% sure how to go about incrementally adding data to an EPT data set. I see merge in the docs and that this should not be used to merge un-related EPT 'files'.

If both EPT's are made with data for same body, in the same SRS, are they related and I should use merge? Or is there another mechanism to incrementally add data (maybe it's as simple as entwine build and the an s3 sync (though this seems wasteful?).

Thanks for any suggestions.

[connormanning]

You can use the same output for multiple builds, with some important limitations:

• The bounds of your dataset must be fully specified by your first build, either explicitly or implicitly. For example if you build some data with a bounds of [0, 0, 0, 5, 5, 5] and then later add data with point [7, 7, 7], it will be discarded since it's outside of the bounds. On your first build, if you know the entire bounds of all your future data, you can run build --bounds [...] to set the bounds such that all of your future data will be accepted. If your initial data happens to encapsulate all of the bounds you care about, then you can just reuse the output without any additional hassle.
• Projections must be the same, so make sure your data is consistent or that you are always using --reprojection to the same target.

merge is never applicable in this use-case, it is only applicable when subsets have been created from the same data in multiple builds.

Files should also be de-duplicated by their filename within Entwine. So this common use-case should work:

entwine build -i my-data/ -o output

Then, let's say a new file is added to my-data/. Now, running this exact same command should only add the new file.

If anything described here does not work, feel free to open an issue with specifics.