Hope you are havingt a great holiday! Thought Id add this to track as we discussed and you thught simple enough to do.
As discussed would be great to be able to invoke these adhoc DML commands when required. The use-case can include things like hygene / cleaning up data controlled via a periodic spark jobs (e.g periodic removal of any duplicates, updating a corrupted column (e.g bq_load_timestamp from a previous bug etc). The
They dont return a DataFrame in the classical sense, however these are more for jobs that help to maintain data quality.
samelamin
commented
7 years ago
Hi Sam,
Hope you are havingt a great holiday! Thought Id add this to track as we discussed and you thught simple enough to do.
As discussed would be great to be able to invoke these adhoc DML commands when required. The use-case can include things like hygene / cleaning up data controlled via a periodic spark jobs (e.g periodic removal of any duplicates, updating a corrupted column (e.g bq_load_timestamp from a previous bug etc). The
They dont return a DataFrame in the classical sense, however these are more for jobs that help to maintain data quality.
Cheers mate!