clab / dynet

DyNet: The Dynamic Neural Network Toolkit
Apache License 2.0
3.43k stars 704 forks source link

Add support to release Linux aarch64 wheels #1651

Open odidev opened 2 years ago

odidev commented 2 years ago

Problem

On aarch64, ‘pip install dyNET’ builds the wheels from source code and is giving below error:

[ 71%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-random.cc.o  
  /tmp/pip-install-u5zcjrap/dyNET/dynet/mem.cc:13:10: fatal error: mm_malloc.h: No such file or directory  
     13 | #include <mm_malloc.h>  
        |          ^~~~~~~~~~~~~  
  compilation terminated.  
  make[2]: *** [dynet/CMakeFiles/dynet.dir/build.make:297: dynet/CMakeFiles/dynet.dir/mem.cc.o] Error 1  
  make[2]: *** Waiting for unfinished jobs....  
  /tmp/pip-install-u5zcjrap/dyNET/dynet/lstm.cc: In member function ‘void dynet::SparseLSTMBuilder::set_sparsity(float)’:  
  /tmp/pip-install-u5zcjrap/dyNET/dynet/lstm.cc:686:19: warning: comparison of integer expressions of different signedness: ‘int’ and ‘unsigned int’ [-Wsign-compare]  
    686 |     for (int i=0;i<layers;i++){  
        |                  ~^~~~~~~  
  /tmp/pip-install-u5zcjrap/dyNET/dynet/expr.cc: In function ‘dynet::Expression dynet::strided_select(const dynet::Expression&, const std::vector<int>&, const std::vector<int>&, const std::vector<int>&)’:  
  /tmp/pip-install-u5zcjrap/dyNET/dynet/expr.cc:201:74: warning: comparison of integer expressions of different signedness: ‘const value_type’ {aka ‘const int’} and ‘unsigned int’ [-Wsign-compare]  
    201 |   for(unsigned d=0;d<range_to.size() && d<x.dim().nd;d++){ if(range_to[d]!=x.dim()[d]) inplaced = false; }  
  make[1]: *** [CMakeFiles/Makefile2:116: dynet/CMakeFiles/dynet.dir/all] Error 2  
  make: *** [Makefile:130: all] Error 2
  /tmp/pip-build-env-sbod7r54/overlay/lib/python3.8/site-packages/setuptools/dist.py:516: UserWarning: Normalizing 'v2.1.2' to '2.1.2'  
    warnings.warn(tmpl.format(**locals()))  
  /tmp/pip-build-env-sbod7r54/overlay/lib/python3.8/site-packages/setuptools/dist.py:757: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead  
    warnings.warn(  
  error: /usr/bin/make -j 48
  ----------------------------------------  
  ERROR: Failed building wheel for dyNET  
Failed to build dyNET
ERROR: Could not build wheels for dyNET which use PEP 517 and cannot be installed directly

Resolution

On aarch64, ‘pip install dyNET’ should download the wheels from pypi.

I have modified the code in order to add support for Linux aarch64 wheel. Aarch64 wheels is building successfully but test cases are getting hanged at a point as below:

python test.py 
[dynet] random seed: 238741976 
[dynet] allocating memory: 512MB 
[dynet] memory allocation done. 
....Reading clusters from cluster_file.txt ... 
Read 10 words in 5 clusters (0 singleton clusters) 
........................ 
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself. 
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received 
The build has been terminated

Commit Link - https://github.com/odidev/dynet/commit/9fc79c50d9f723cd899622290c881edb73dd494f

Travis Link - https://app.travis-ci.com/github/odidev/dynet/builds/249269741

@Team Please let me know your interest in releasing Linux aarch64 wheels. To start with, can I get some suggestions on why test cases are getting stuck at a point?