Enable Dataflow managed service for Python tests

ahmedabu98 commented 1 week ago

Enable Dataflow managed backend service for python managed tests

github-actions[bot] commented 1 week ago

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label python. R: @damccorm for label build.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

damccorm commented 6 days ago

Of the failing tests:

https://github.com/apache/beam/actions/runs/11874699399/job/33142752731?pr=33134 might be concerning
the linter should be resolved by the changes I suggested
The remaining 2 should be fixed by https://github.com/apache/beam/pull/33148

damccorm commented 6 days ago

FAILED apache_beam/transforms/managed_iceberg_it_test.py::ManagedIcebergIT::test_write_read_pipeline - apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
org.apache.beam.sdk.util.UserCodeException: org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist: test_iceberg_write_read.test_1731941722
    at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
    at org.apache.beam.sdk.io.iceberg.WriteUngroupedRowsToFiles$WriteUngroupedRowsToFilesDoFn$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForWindowObservingParDo(FnApiDoFnRunner.java:823)
    at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:348)
    at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:275)
    at org.apache.beam.fn.harness.FnApiDoFnRunner.outputTo(FnApiDoFnRunner.java:1837)
    at org.apache.beam.fn.harness.FnApiDoFnRunner.access$3100(FnApiDoFnRunner.java:145)
    at org.apache.beam.fn.harness.FnApiDoFnRunner$WindowObservingProcessBundleContext.output(FnApiDoFnRunner.java:2240)
    at org.apache.beam.sdk.io.iceberg.AssignDestinations$1.processElement(AssignDestinations.java:66)
    at org.apache.beam.sdk.io.iceberg.AssignDestinations$1$DoFnInvoker.invokeProcessElement(Unknown Source)
    at org.apache.beam.fn.harness.FnApiDoFnRunner.processElementForWindowObservingParDo(FnApiDoFnRunner.java:823)
    at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:348)
    at org.apache.beam.fn.harness.data.PCollectionConsumerRegistry$MetricTrackingFnDataReceiver.accept(PCollectionConsumerRegistry.java:275)
    at org.apache.beam.fn.harness.BeamFnDataReadRunner.forwardElementToConsumer(BeamFnDataReadRunner.java:213)
    at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.multiplexElements(BeamFnDataInboundObserver.java:172)
    at org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:136)
    at org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:550)
    at org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
    at org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist: test_iceberg_write_read.test_1731941722
    at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:53)
    at org.apache.beam.sdk.io.iceberg.RecordWriterManager.lambda$write$0(RecordWriterManager.java:236)
    at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1134)
    at org.apache.beam.sdk.io.iceberg.RecordWriterManager.write(RecordWriterManager.java:233)
    at org.apache.beam.sdk.io.iceberg.WriteUngroupedRowsToFiles$WriteUngroupedRowsToFilesDoFn.processElement(WriteUngroupedRowsToFiles.java:243)

This looks like a real issue

damccorm commented 6 days ago

Failing checks are known flakes/should be fixed by https://github.com/apache/beam/pull/33151

damccorm commented 6 days ago

Oh whoops, merged the wrong pr - meant to merge #33150

apache / beam

Enable Dataflow managed service for Python tests #33134