search

apache / incubator-uniffle

Uniffle is a high performance, general purpose Remote Shuffle Service.
https://uniffle.apache.org/
Apache License 2.0
381 stars 148 forks source link

[Umbrella] Flink Support (Help Wanted) #737

Open jerqi opened 1 year ago

jerqi commented 1 year ago

Code of Conduct

Search before asking

Describe the proposal

Flink Support is complex topic. We set up an issue here. In Flink Community, Flink Shuffle will have much change because of https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage. We should discuss actively with Flink community. Their discussion thread https://lists.apache.org/thread/nwrqd5jtqwks89tbxpcrgto6r2bhdhno

Task list

xx

Are you willing to submit PR?

slfan1989 commented 1 year ago

This document can also be used as a reference.

FLIP-31: Pluggable Shuffle Service

FLIP-148: Introduce Sort-Based Blocking Shuffle to Flink

FLIP-184: Refine ShuffleMaster lifecycle management for pluggable shuffle service framework

FLIP-199: Change some default config values of blocking shuffle for better usability

FLIP-209: Support to run multiple shuffle plugins in one session cluster

FLIP-301: Hybrid Shuffle supports Remote Storage

Vedio: Pluggable Shuffle Service and Unaligned Checkpoints

flink-remote-shuffle

https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks

slfan1989 commented 1 year ago

@jerqi I will follow up the issue with chenglong(Big Data Development Engineer In Lixiang Car).

jerqi commented 1 year ago

Thanks @slfan1989

chenglong-bigdata commented 1 year ago

with me

jerqi commented 1 year ago

with me

Thanks for your help.

pegasas commented 1 year ago

Hi, @jerqi and the team,

I am interested in this project due to investigating Flink batch shuffle service in my previous job.

Would I still be able to join this project?

jerqi commented 1 year ago

Hi, @jerqi and the team,

I am interested in this project due to investigating Flink batch shuffle service in my previous job.

Would I still be able to join this project?

You're welcome. Currently, we have a regular meeting. If you have interest and can Speak Chinese, you can join the Wechat Group, the meeting link is the Wechat Group.

zuston commented 1 year ago

cc @slfan1989 Do you already have a draft implementation? I remember you mentioned that in the online meeting

slfan1989 commented 11 months ago

cc @slfan1989 Do you already have a draft implementation? I remember you mentioned that in the online meeting

@zuston My draft version code has been basically completed (but some details still need to be optimized), I will submit PR to the community as soon as possible.