search

ICRAR / daliuge

The DALiuGE Execution Engine
GNU Lesser General Public License v2.1
26 stars 7 forks source link

Install issues with DALiuGE #17

Closed myxie closed 3 years ago

myxie commented 4 years ago

I have been trying to use the daliuge-translator to do some unrolling/partitioning of files, and have been having issues.

When I install DALiuGE from the main directory, using the local copy cloned from the master branch, it uses an old version of the common/init.py file. However, when I install the components separately (i.e., enter the daliuge-translator/, daliuge-common/, and daliuge-runtime) directories, and install each component separately from there, it works fine and uses the correct libraries.

I think the setup.py file is getting an older version of DALiuGE (maybe from PyPi?), and this is causing the conflicts.

Installing them all separately stops this from being a 'block', however.

james-strauss-uwa commented 4 years ago

@rtobar My knowledge of PyPi is pretty limited, is there something I need to do to make sure PyPI serves the latest version of the code.

rtobar commented 4 years ago

@myxie "When I install DALiuGE from the main directory" -> I think that's the problem, but this is because the documentation is not very clear now about this I think.

As you found out, when you install each sub-directory then everything works fine, because that's the intended method of installation when working locally. Each subdirectory contains a python package with the same name (daliuge-common, daliuge-translator and daliuge-runtime). We upload each of these to PyPI, so one can also install each of them via pip without having to download the git repo.

The top-level directory of the repository contains a setup.py file for yet a fourth python package (daliuge) that simply declares the other three as its dependencies. When you try to install this one, the other three dependencies are automatically pulled. Not from the filesystem though, but rather from PyPI. This is just how setuptools/pip work: when there is a dependency, it's fetched from the PyPI repositories.

This is probably the problem you are seeing, and is the result of a confusing top-level directory structure. What we can do on our end is to clarify this, maybe with a better README file, and maybe also moving away the top-level setup.py into a sub-directory to avoid confusions.

@james-strauss-uwa if you need to upload a new version of the code to PyPI let me know. I can grant you permissions (you will need a PyPI account) and I can show how to do it.

myxie commented 3 years ago

@rtobar, @james-strauss-uwa I am going to close this issue, as @rtobar's discussion and the updated DALiuGE docs are enough to 'fix' this issue.

Many thanks for your help!