[Hadoop compression support] Can Hadoop support Snappy and Lzo compression

datavane / datasophon

The next generation of cloud-native big data management expert , Aims to help users rapidly build stable, efficient, and scalable cloud-native platforms for big data.

https://datasophon.github.io/datasophon-website/

Apache License 2.0

1.13k stars 393 forks source link

[Hadoop compression support] Can Hadoop support Snappy and Lzo compression #271

Open zqwangadv opened 1 year ago

zqwangadv commented 1 year ago

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description

Hadoop能否编译成支持LZO和Snappy 二进制包

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

github-actions[bot] commented 1 year ago

Search before asking

[X] I had searched in the feature and found no similar feature requirement.

Description

Hadoop能否编译成支持LZO和Snappy 二进制包

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

github-actions[bot] commented 1 year ago

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.

datasophon commented 1 year ago

Yes ，Can you contribute this?

BsoBird commented 1 year ago

@datasophon I didn't find any HADOOP project address under datasophon. The DDP code does not seem to be publicly available either. It looks like there is no way to submit the code.

datasophon commented 1 year ago

@datasophon I didn't find any HADOOP project address under datasophon. The DDP code does not seem to be publicly available either. It looks like there is no way to submit the code.

We use open-source Hadoop to build DDP, so this issue requires encapsulating the Hadoop installation package and adding relevant jar packages

zqwangadv commented 1 year ago

I have compile the hadoop to support snappy and lzo compression, but I do not know how to encapsulate it so that it can work with DDP.

datasophon commented 1 year ago

Thank you for your contribution.You can send me the Hadoop installation package offline, and I will put you in the contributor list

I have compile the hadoop to support snappy and lzo compression, but I do not know how to encapsulate it so that it can work with DDP.

Thank you for your contribution.You can send me the Hadoop installation package offline and I will put you in the contributor list

zqwangadv commented 1 year ago

You can download the installation package from the following link: https://pan.baidu.com/s/1HnrYFFDO9jN-DSjO9K53nw . The extraction code is: 2r2t.