Mac m1机器上python 3.10安装paddle-bfloat报错

[pangyoki]

问题描述 Issue Description

如题，在Mac m1机器上，python3.10（universal版本）单独安装paddle-bfloat会报错，如下图。初步测试linux、windows、非m1的mac机器上，python3.10可以正常安装paddle-bfloat。

看起来是pypi上没有相应的预编译包，默认去拉tar.gz本地编译安装，可能暂时不支持mac m1环境下的编译安装。

paddle准备支持python3.10，依赖paddle-bfloat对mac m1机器python3.10的支持，希望能高优解决这个问题。

版本&环境信息 Version & Environment Information

Mac m1机器，python3.10 universal （https://www.python.org/ftp/python/3.10.4/python-3.10.4-macos11.pkg）

[paddle-bot-old[bot]]

您好，我们已经收到了您的问题，会安排技术人员尽快解答您的问题，请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时，您也可以通过查看官网API文档、常见问题、历史Issue、AI社区来寻求解答。祝您生活愉快～

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the API，FAQ，Github Issue and AI community to get the answer.Have a nice day!

[Aganlengzi]

@lidanqing-intel @jakpiase hi, in addition, please upload the pre-compiled packages of other tags to avoid local compiling, I think we can refer to Pillow https://pypi.org/project/Pillow/#files

[jakpiase]

Hi @Aganlengzi, @pangyoki. I have uploaded additional wheels for paddle_bfloat v0.1.2 into pypi: https://pypi.org/project/paddle-bfloat/0.1.2/#files. Mac scenario with python 3.10 is also included here, so I think that your problem should be fixed. Please let me know if it resolved your issue, thanks

[Aganlengzi]

Hi @Aganlengzi, @pangyoki. I have uploaded additional wheels for paddle_bfloat v0.1.2 into pypi: https://pypi.org/project/paddle-bfloat/0.1.2/#files. Mac scenario with python 3.10 is also included here, so I think that your problem should be fixed. Please let me know if it resolved your issue, thanks

@jakpiase Thank you very much! I checked in my Mac with m1 and python 3.10, it succeed:

[Aganlengzi]

@jakpiase Hi, which numpy version is used in your pre-compiled wheels? could it align with Paddle's requirements? Paddle requires numpy>=1.13 in https://github.com/PaddlePaddle/Paddle/blob/develop/python/requirements.txt but in some case, it reports incompatible problem：

2022-04-24 12:38:26 1/1 Test #1264: test_python_bf16_numpy_datatype ...***Failed    2.46 sec
2022-04-24 12:38:26 RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe
2022-04-24 12:38:26 ImportError: numpy.core._multiarray_umath failed to import
2022-04-24 12:38:26 ImportError: numpy.core.umath failed to import

https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/5478824/job/14350996

[jakpiase]

Hi @Aganlengzi, I have uploaded some wheels with old numpy versions, sorry for that. I have deleted them, now pypi repository should contain only versions which were built with numpy>=1.13

[Aganlengzi]

Hi @Aganlengzi, I have uploaded some wheels with old numpy versions, sorry for that. I have deleted them, now pypi repository should contain only versions which were built with numpy>=1.13

Hi, @jakpiase thanks for quick reply, but the message seems saying that the numpy version you compiled with is higher than installed one(0xf > 0xe)? if this is true, I think you may compile wheels with lower numpy version(at most 1.13)? I am not sure about that.

Tried locally with wheel: paddle_bfloat-0.1.2-cp37-cp37m-manylinux_2_27_x86_64.whl

Another try with newest numpy and wheel: numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

[Aganlengzi]

@lidanqing-intel @jakpiase hi, there is no arm64 package in https://pypi.org/project/paddle-bfloat/0.1.2/#files, would you please help upload pre-built arm64 package?

[jakpiase]

Hi @Aganlengzi, I have uploaded arm64 wheels for python3.8-3.10. Just FYI I am currently working on newer version of paddle_bfloat package and it should be finished soon. All future versions of paddle_bfloat will cover all types of wheels that are currently supported. New wheels for older linux systems will be added as well. I have managed to fix issues associated with different numpy versions between paddle_bfloat and client system. In future versions paddle_bfloat wheels will match all numpy wheels, so following combinations of python and numpy will be supported:

Python version	Numpy versions
3.6	1.13-1.19
3.7	1.15-1.21
3.8	1.18-1.22
3.9	1.20-1.22
3.10	1.22

[Aganlengzi]

Have seen precompiled packages, reopen if any related issues.

[shiyutang]

linux python3.8 pip-22.1 安装paddle-bfloat=0.1.2 报错：

1. python3.8 -m pip install paddle-bfloat==0.1.2 --no-cache
2. 网页上下载对应版本，不能安装：

[Aganlengzi]

@lidanqing-intel @jakpiase hi, paddle-bfloat==0.1.2 is depended by 2.3, it is very important to keep as many as possible pre-built wheels, please help check, thanks. BTW, In @shiyutang‘s env, package with the following tag is ok, but can not be found in this site:https://pypi.org/project/paddle-bfloat/0.1.2/#files

[shiyutang]

这样之后，就可以了

[lidanqing-intel]

still fixing paddle_bfloat package CIs, compilation failures are fixed, but there is a problem while building paddlepaddle-gpu package with my changes

[lidanqing-intel]

@jakpiase Please make sure 0.1.2 has as many tags as possible because paddle2.3 is using paddle_bfloat16 0.1.2. Thanks!

[jakpiase]

@Aganlengzi, shiyutang's wheel cannot be found on https://pypi.org/project/paddle-bfloat/0.1.2/#files, because it was deleted by me, since I have built it with newest numpy version and it was not compatible with older ones which was causing errors listed by you above. Today I have deleted most of the broken wheels from paddle_bfloat v0.1.2 andm I have added more new wheels, to cover a wider range of supported machines. The recently added wheels were built with manylinux1 standard, which is the oldest possible(supports distros like Centos 5), so all linux machines should work properly now. I have tested that locally, and linux wheels for all python version supported by PaddlePaddle should work correctly without issues like RuntimeError: module compiled against API version 0xf but this version of numpy is 0xe. Although I have fixed wheels for linux machines for v0.1.2, wheels published for windows and macos still won't work 100% correctly, but I cannot replace them, because PyPi doesn't allow publishing a wheel which have the same name as previously added one because of security reasons, so the only way to fix all current issues related to paddle_bfloat package is to upgrade it inside PaddlePaddle to version 0.1.7, which is the newest and is fully tested by me and has all wheels working correctly.

All currently published wheels for the two important versions of paddle_bfloat:

• v0.1.2 wheels: https://pypi.org/project/paddle-bfloat/0.1.2/#files
• v0.1.7 wheels: https://pypi.org/project/paddle-bfloat/0.1.7/#files

[yaomichael]

Notes from 5/20 meeting: @jiangjiajun will arrange test for Paddle-bfloat 0.1.7 and upgrade PaddlePaddle with it.