search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.55k stars 1.6k forks source link

Update taxi pipeline example with latest beam with S3 support #3185

Closed Jeffwan closed 5 months ago

Jeffwan commented 4 years ago

Improvement:

We have a lot of pain in https://github.com/kubeflow/pipelines/issues/596. AWS user have to use EFS or other ReadWriteMany storage for taxi examples. S3 FileSystem (Python) has been implemented in Apache-Beam 2.19.0, then we can leverage this to simplify taxi example.

Jeffwan commented 4 years ago

/cc @PatrickXYS will help on the example update.

numerology commented 4 years ago

Cool! Thanks for willing to help

Bobgy commented 4 years ago

/assign @PatrickXYS

k8s-ci-robot commented 4 years ago

@Bobgy: GitHub didn't allow me to assign the following users: PatrickXYS.

Note that only kubeflow members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to [this](https://github.com/kubeflow/pipelines/issues/3185#issuecomment-594344773): >/assign @PatrickXYS Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
PatrickXYS commented 4 years ago

@Jeffwan I still remember last time we talked about this issue, we'll figure out what Apache Beam version will TFX team use, and if they're not planning using 2.19 version, we should talk with them to see how it goes.

/assign @PatrickXYS

Jeffwan commented 4 years ago

@PatrickXYS vanila TFX still use beam 2.17, Let me file a ticket in TFX to upgrade to 2.19. This is currently blocked. We probably can revisit this later

brychcy commented 4 years ago

It looks like TFX is currently not compatible with Apache Beam 2.19, see https://github.com/tensorflow/tfx/issues/1219

Jeffwan commented 4 years ago

It looks like TFX is currently not compatible with Apache Beam 2.19, see tensorflow/tfx#1219

Yes. It maybe not a simple version upgrade. Let's track in the issue there. I create another one for upgrade https://github.com/tensorflow/tfx/issues/1446

PatrickXYS commented 4 years ago

The latest TFX release still uses beam 2.17.0 . So the issue is still blocking there.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 4 years ago

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

PatrickXYS commented 4 years ago

/reopen

k8s-ci-robot commented 4 years ago

@PatrickXYS: Reopened this issue.

In response to [this](https://github.com/kubeflow/pipelines/issues/3185#issuecomment-685284721): >/reopen > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

PatrickXYS commented 3 years ago

Not yet fixed

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

goswamig commented 2 years ago

not yet fixed

rimolive commented 5 months ago

Closing this issue. No activity for a while.

/close

google-oss-prow[bot] commented 5 months ago

@rimolive: Closing this issue.

In response to [this](https://github.com/kubeflow/pipelines/issues/3185#issuecomment-2016817474): >Closing this issue. No activity for a while. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.