Not able to create a fat file

rbhatia46 commented 4 years ago

Hi, I am using the following command -

python setup.py pyassembly

Then, I am navigating the pyassembly_dist/ folder, where I have an egg file, now when I am using this egg file, it doesnt contain all the dependent dependencies, in short, it doesnt create a fat egg file, am I missing anything?

alekseyig commented 4 years ago

@rbhatia46, please follow the README, it tells how to install and run it. Let me know if you still have problems.

rbhatia46 commented 4 years ago

@alekseyig as I mentioned, I am using the same command in README after installing to generate the egg file, but it doesn't create a fat egg, it only creates for that dependency, and not all other transitive dependencies, could you please help me with this?

alekseyig commented 4 years ago

@rbhatia46

could you please help me with this?

sure, can you send me the call and the output when you run pyassembly command? can you share your setup.py?

rbhatia46 commented 4 years ago

Sure. Running this command -

python setup.py pyassembly

Here is the output -

running pyassembly
Collecting Flask==1.0.2
  Using cached Flask-1.0.2-py2.py3-none-any.whl (91 kB)
Collecting hyperas==0.4
  Using cached hyperas-0.4.tar.gz (6.6 kB)
Collecting Keras==2.2.2
  Using cached Keras-2.2.2-py2.py3-none-any.whl (299 kB)
Collecting Keras-Applications==1.0.4
  Using cached Keras_Applications-1.0.4-py2.py3-none-any.whl (43 kB)
Collecting Keras-Preprocessing==1.0.2
  Using cached Keras_Preprocessing-1.0.2-py2.py3-none-any.whl (26 kB)
Collecting numpy==1.14.5
  Using cached numpy-1.14.5-cp36-cp36m-manylinux1_x86_64.whl (12.2 MB)
Processing /root/.cache/pip/wheels/4e/a7/1b/0c19c7e64c0c3fe3dc0e070203f7a1e4e2455080bd4c6b7029/pyspark-2.4.0-py2.py3-none-any.whl
Collecting six==1.11.0
  Using cached six-1.11.0-py2.py3-none-any.whl (10 kB)
Collecting tensorflow==1.15.2
  Using cached tensorflow-1.15.2-cp36-cp36m-manylinux2010_x86_64.whl (110.5 MB)
Collecting networkx==1.1
  Using cached networkx-1.1.tar.gz (562 kB)
running egg_info
creating elephas.egg-info
writing elephas.egg-info/PKG-INFO
writing dependency_links to elephas.egg-info/dependency_links.txt
writing requirements to elephas.egg-info/requires.txt
writing top-level names to elephas.egg-info/top_level.txt
writing manifest file 'elephas.egg-info/SOURCES.txt'
reading manifest file 'elephas.egg-info/SOURCES.txt'
writing manifest file 'elephas.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/pyassembly
running install_lib
running build_py
creating build/lib
creating build/lib/elephas
copying elephas/__init__.py -> build/lib/elephas
copying elephas/hyperparam.py -> build/lib/elephas
copying elephas/spark_model.py -> build/lib/elephas
copying elephas/ml_model.py -> build/lib/elephas
copying elephas/worker.py -> build/lib/elephas
copying elephas/dl4j.py -> build/lib/elephas
creating build/lib/docs
copying docs/__init__.py -> build/lib/docs
copying docs/autogen.py -> build/lib/docs
creating build/lib/elephas/ml
copying elephas/ml/__init__.py -> build/lib/elephas/ml
copying elephas/ml/params.py -> build/lib/elephas/ml
copying elephas/ml/adapter.py -> build/lib/elephas/ml
creating build/lib/elephas/parameter
copying elephas/parameter/server.py -> build/lib/elephas/parameter
copying elephas/parameter/__init__.py -> build/lib/elephas/parameter
copying elephas/parameter/client.py -> build/lib/elephas/parameter
creating build/lib/elephas/mllib
copying elephas/mllib/__init__.py -> build/lib/elephas/mllib
copying elephas/mllib/adapter.py -> build/lib/elephas/mllib
creating build/lib/elephas/utils
copying elephas/utils/sockets.py -> build/lib/elephas/utils
copying elephas/utils/rdd_utils.py -> build/lib/elephas/utils
copying elephas/utils/serialization.py -> build/lib/elephas/utils
copying elephas/utils/functional_utils.py -> build/lib/elephas/utils
copying elephas/utils/__init__.py -> build/lib/elephas/utils
copying elephas/utils/rwlock.py -> build/lib/elephas/utils
copying elephas/utils/notebook_utils.py -> build/lib/elephas/utils
creating build/lib/elephas/java
copying elephas/java/java_classes.py -> build/lib/elephas/java
copying elephas/java/ndarray.py -> build/lib/elephas/java
copying elephas/java/__init__.py -> build/lib/elephas/java
copying elephas/java/adapter.py -> build/lib/elephas/java
creating build/bdist.linux-x86_64/pyassembly
creating build/bdist.linux-x86_64/pyassembly/elephas
creating build/bdist.linux-x86_64/pyassembly/elephas/ml
copying build/lib/elephas/ml/__init__.py -> build/bdist.linux-x86_64/pyassembly/elephas/ml
copying build/lib/elephas/ml/params.py -> build/bdist.linux-x86_64/pyassembly/elephas/ml
copying build/lib/elephas/ml/adapter.py -> build/bdist.linux-x86_64/pyassembly/elephas/ml
copying build/lib/elephas/__init__.py -> build/bdist.linux-x86_64/pyassembly/elephas
copying build/lib/elephas/hyperparam.py -> build/bdist.linux-x86_64/pyassembly/elephas
creating build/bdist.linux-x86_64/pyassembly/elephas/parameter
copying build/lib/elephas/parameter/server.py -> build/bdist.linux-x86_64/pyassembly/elephas/parameter
copying build/lib/elephas/parameter/__init__.py -> build/bdist.linux-x86_64/pyassembly/elephas/parameter
copying build/lib/elephas/parameter/client.py -> build/bdist.linux-x86_64/pyassembly/elephas/parameter
copying build/lib/elephas/spark_model.py -> build/bdist.linux-x86_64/pyassembly/elephas
creating build/bdist.linux-x86_64/pyassembly/elephas/mllib
copying build/lib/elephas/mllib/__init__.py -> build/bdist.linux-x86_64/pyassembly/elephas/mllib
copying build/lib/elephas/mllib/adapter.py -> build/bdist.linux-x86_64/pyassembly/elephas/mllib
copying build/lib/elephas/ml_model.py -> build/bdist.linux-x86_64/pyassembly/elephas
creating build/bdist.linux-x86_64/pyassembly/elephas/utils
copying build/lib/elephas/utils/sockets.py -> build/bdist.linux-x86_64/pyassembly/elephas/utils
copying build/lib/elephas/utils/rdd_utils.py -> build/bdist.linux-x86_64/pyassembly/elephas/utils
copying build/lib/elephas/utils/serialization.py -> build/bdist.linux-x86_64/pyassembly/elephas/utils
copying build/lib/elephas/utils/functional_utils.py -> build/bdist.linux-x86_64/pyassembly/elephas/utils
copying build/lib/elephas/utils/__init__.py -> build/bdist.linux-x86_64/pyassembly/elephas/utils
copying build/lib/elephas/utils/rwlock.py -> build/bdist.linux-x86_64/pyassembly/elephas/utils
copying build/lib/elephas/utils/notebook_utils.py -> build/bdist.linux-x86_64/pyassembly/elephas/utils
copying build/lib/elephas/worker.py -> build/bdist.linux-x86_64/pyassembly/elephas
creating build/bdist.linux-x86_64/pyassembly/elephas/java
copying build/lib/elephas/java/java_classes.py -> build/bdist.linux-x86_64/pyassembly/elephas/java
copying build/lib/elephas/java/ndarray.py -> build/bdist.linux-x86_64/pyassembly/elephas/java
copying build/lib/elephas/java/__init__.py -> build/bdist.linux-x86_64/pyassembly/elephas/java
copying build/lib/elephas/java/adapter.py -> build/bdist.linux-x86_64/pyassembly/elephas/java
copying build/lib/elephas/dl4j.py -> build/bdist.linux-x86_64/pyassembly/elephas
creating build/bdist.linux-x86_64/pyassembly/docs
copying build/lib/docs/__init__.py -> build/bdist.linux-x86_64/pyassembly/docs
copying build/lib/docs/autogen.py -> build/bdist.linux-x86_64/pyassembly/docs
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/ml/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/ml/params.py to params.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/ml/adapter.py to adapter.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/hyperparam.py to hyperparam.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/parameter/server.py to server.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/parameter/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/parameter/client.py to client.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/spark_model.py to spark_model.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/mllib/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/mllib/adapter.py to adapter.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/ml_model.py to ml_model.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/utils/sockets.py to sockets.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/utils/rdd_utils.py to rdd_utils.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/utils/serialization.py to serialization.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/utils/functional_utils.py to functional_utils.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/utils/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/utils/rwlock.py to rwlock.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/utils/notebook_utils.py to notebook_utils.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/worker.py to worker.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/java/java_classes.py to java_classes.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/java/ndarray.py to ndarray.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/java/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/java/adapter.py to adapter.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/elephas/dl4j.py to dl4j.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/docs/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/pyassembly/docs/autogen.py to autogen.cpython-36.pyc
creating build/bdist.linux-x86_64/pyassembly/EGG-INFO
copying elephas.egg-info/PKG-INFO -> build/bdist.linux-x86_64/pyassembly/EGG-INFO
copying elephas.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/pyassembly/EGG-INFO
copying elephas.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/pyassembly/EGG-INFO
copying elephas.egg-info/not-zip-safe -> build/bdist.linux-x86_64/pyassembly/EGG-INFO
copying elephas.egg-info/requires.txt -> build/bdist.linux-x86_64/pyassembly/EGG-INFO
copying elephas.egg-info/top_level.txt -> build/bdist.linux-x86_64/pyassembly/EGG-INFO
creating pyassembly_dist
creating 'pyassembly_dist/elephas-0.4.3-py3.6.egg' and adding 'build/bdist.linux-x86_64/pyassembly' to it

And this is the setup.py file.

alekseyig commented 4 years ago

@rbhatia46 , sorry forgot to ask, what version of Python are you using?

rbhatia46 commented 4 years ago

Python 3.6

alekseyig commented 4 years ago

@rbhatia46,

that is what i did:

git checked out https://github.com/maxpumperla/elephas

modified the requrements.txt file to add pyassembly (i commented out the networkx because the build was failing on it, but for our test it does not matter)

$ cat requirements.txt
Flask==1.0.2
hyperas==0.4
Keras==2.2.2
Keras-Applications==1.0.4
Keras-Preprocessing==1.0.2
numpy==1.14.5
pyspark==2.4.0
six==1.11.0
tensorflow==1.15.2
#networkx==1.1
pydl4j>=0.1.3
pyassembly

ran pip install -r requirements.txt

ran python setup.py --help-commands to make sure pyassembly got installed

$ python setup.py --help-commands
Standard commands:
build             build everything needed to install
build_py          "build" pure Python modules (copy to build directory)
...
Extra commands:
bdist_wheel       create a wheel distribution
alias             define a shortcut to invoke one or more commands
...
pyassembly        create an egg or zip distribution with all its transitive dependencies

ran the command python setup.py pyassembly
$ cd pyassembly_dist
ran ls
```
$ ls
elephas-0.4.3-py3.6.egg
```
the egg file is a zip file with some extras, so we can do
```
$ mv elephas-0.4.3-py3.6.egg elephas-0.4.3-py3.6.zip
```
ran
```
$ unzip elephas-0.4.3-py3.6.zip
```
compared the list of packages unziped from the zip(egg) file above with pip list and i see all the packages installed with pip in current dir

Can you do similar steps and see if it would help? So far i am not sure what the problem you are facing.

rbhatia46 commented 4 years ago

Hi, The steps mentioned above works fine, thanks for the help.

alekseyig / PyAssembly

Not able to create a fat file #5