CDP: Add BufferedStorageBackend usage to Stellar developer docs

urvisavla commented 1 week ago

What problem does your feature solve?

In https://github.com/stellar/stellar-docs/issues/1012 we added Galexie to Stellar’s public developer docs. This includes an introduction to Galexie and a guide on how to run it—essentially an admin guide that focuses on the publisher side. However, we also want to include documentation for the consumer side, specifically on how to use BufferedStorageBackend to read data from the data lake.

What would you like to see?

CDP Development - Explain how data pipeline arch is preferred for ingestion and how datastore of pre-computed ledger metadata provides the origin in a pipeline.
cover the go SDK packages that can be used for pipeline development: support/datastore to use GCSDataStore, ingest/ledgerbackend to use BufferedStorageBackend, ingest/cdp to use helper functions like PublishFromBufferedStorageBackend

@sreuland has already created a sample ingestion pipeline using BufferedStorageBackend. We could either link to this pipeline from the main Galexie docs/admin guide or create a new section that explains BufferedStorageBackend in detail covering its parameters and then link to the ingestion pipeline example from there.

What alternatives are there?

sreuland commented 1 week ago

Other public copy on CDP consumption side is actually referencing the newer functional wrapper around BSB, cdp.ApplyLedgerMetadata(), rather than using BSB directly, can this be leveraged here to simplify the docs needed? i.e. document preference towards usage of ApplyLedgerMetadata() and it's companion helper function, DefaultBufferedStorageBackendConfig() , ideally this equates to less writeup, maybe briefly mention at high level how it encapsulates BSB usage so client doesn't have to?

urvisavla commented 1 week ago

Other public copy on CDP consumption side is actually referencing the newer functional wrapper around BSB, cdp.ApplyLedgerMetadata(), rather than using BSB directly, can this be leveraged here to simplify the docs needed? i.e. document preference towards usage of ApplyLedgerMetadata() and it's companion helper function, DefaultBufferedStorageBackendConfig() , ideally this equates to less writeup, maybe briefly mention at high level how it encapsulates BSB usage so client doesn't have to?

Yes, that makes more sense.

stellar / stellar-docs