When dealing with large scales of data, figuring just how best to whitelist and what needs to come over is a lot of trial and error.
One issue I see is; I need to do a full run of everything before I can get a spitout of my error (because the real-time error is "Reached limit of error for a table". Now, I can increase the verbosity of the logging and learn earlier of what is going on, but then the noise-to-signal ratio goes way high.
If you could just output the Error when you hit it, or quit the ETL at the first error. Then it would be far more effective for me to round trip.
When dealing with large scales of data, figuring just how best to whitelist and what needs to come over is a lot of trial and error.
One issue I see is; I need to do a full run of everything before I can get a spitout of my error (because the real-time error is "Reached limit of error for a table". Now, I can increase the verbosity of the logging and learn earlier of what is going on, but then the noise-to-signal ratio goes way high.
If you could just output the Error when you hit it, or quit the ETL at the first error. Then it would be far more effective for me to round trip.
Thanks!