Open burtenshaw opened 5 days ago
I'd rather have the records ingested and pushed in batches and have an easy way to identify those that threw an error, fix them and try to import those again.
Otherwise it can take ages until I see any records in my dataset.
This PR supports mapping incoming columns/keys to dataset attributes in these two ways:
_ingest_records
methods so that mapping is performed once before the ingestion loop instead of during.This PR also optimises the
log
method so that it takes less time and is easier to work with:How Has This Been Tested
Checklist