Support ingesting SST files generated by a live DB

[cbi42]

Summary: ... to enable use cases like using RocksDB to merge sort data for ingestion. A new file ingestion option IngestExternalFileOptions::from_live_db is introduced to allows users to ingest SST files generated by live DBs instead of SstFileWriter. For now this only works if the SST files being ingested have zero as their largest sequence number.

Since the feature if not forward-compatible, marked the option as experimental for now in case we want a more explicit opt-in, e.g., introduce MANIFEST versioning and require setting a new MANIFEST version.

Main changes needed to enable this:

• ignore CF id mismatch during ingestion
• ignore the missing external file version table property
• add a new field ignore_seqno_in_file to FileMetaData (and persisted in MANIFEST) to let table reader know to use largest sequence number as the global sequence number for reading the file.

Rest of the change is mostly new unit tests and stress test changes.

A previous attempt is in #5602.

Test plan:

• new unit tests
• stress test: add a variation of TestIngestExternalFile() enabled with ingest_external_file_one_in that creates the file in a temporary DB.

[facebook-github-bot]

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cbi42 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cbi42 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@cbi42 has updated the pull request. You must reimport the pull request before landing.

[cbi42]

Since the feature if not forward-compatible, marked the option as experimental for now in case we want a more explicit opt-in, e.g., introduce MANIFEST versioning and require setting a new MANIFEST version.

[facebook-github-bot]

@cbi42 has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@cbi42 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.