Ingestion-as-a-library

[2opremio]

We should allow users to create their own core-ingestion pipelines.

High level tasks:

• [x] Make exp/ingest public
• [x] Keep internal/expingest private, but audit it:
  • [x] https://github.com/stellar/go/issues/2851 (expingest -> ingest rename issue here - bartek)
• [ ] Make sure developer experience is good:
  • [x] https://github.com/stellar/go/pull/3318 Rename SingleLedgerStateReader
  • [x] https://github.com/stellar/go/pull/3324 Get rid of io subpackage and move the code to the top level ingest package
  • [x] https://github.com/stellar/go/pull/3330 Get rid of errors subpackage and move StateError to top level ingest package. Also consider removing ingest.EOF alias.
  • [x] #3332 Rename LedgerEntryChangeCache to something which is more indicative of the operation it's performing (compating / coalescing changes) (@2opremio )
  • [x] ~either make mock implementations private or move them to separate testing package~ (skipped, see comment below)
  • [x] #3329 conform implementations to their respective interfaces
  • [x] #3328 move Range to its own file
  • [x] #3333 consider moving StateVerifier to horizon/internal ingestion package (@2opremio)
  • [x] https://github.com/stellar/go/pull/3335 Investigate removing GenesisLedgerStateReader (@tamirms)
  • [x] #3337 consider moving processor interface to horizon/internal ingestion package (@2opremio)
  • [x] https://github.com/stellar/go/pull/3339 Consider removing HistoryArchiveBackend
  • [x] Check documentation
  • [x] #3320 Write examples and a tutorial (piggyback on bartek's ingestion code for the cloudwatch dashboard?) (@Shaptic)
  • [x] ~Look into what the pipeline from the Infra team, dumping info into BigQuery (the library should make their use case possible and simple).~ @tamirms Update https://github.com/stellar/stellar-etl to use the newest github.com/go/ingest (after discussion with @nikhilsaraf)

[Shaptic]

With regards to the following item:

either make mock implementations private or move them to separate testing package

This may not be possible. Enumerating the approaches:

• making mocked objects private: This involves hiding MockChangeProcessor and MockChangeReader. However, doing this results in the horizon/internal/ingest package breaking, since it also uses these mocked objects within its tests.
• move tests to a testing package: This means moving the above two objects into a testing/ subdirectory. However, doing this (and making the necessary changes e.g. package ingest -> package testing) means that the references to the top-level ingest library (like ingest.ChangeReader, see mock_change_reader.go) now cause a circular reference, erroring out with:

$ go test
# github.com/stellar/go/ingest
package github.com/stellar/go/ingest (test)
    imports github.com/stellar/go/ingest/testing
    imports github.com/stellar/go/ingest: import cycle not allowed in test
FAIL    github.com/stellar/go/ingest [setup failed]

[ire-and-curses]

This was completed and released as stellar-ingest-v1.0.0.