Readme and Wiki: Update for FreeCAD integration of PyrateWorkbench

[joha2]

• update usage to be compatible with new-style workbenches
• update how to install AppImage of 0.19 FC and activate pyrate workbench
• update wiki

@looooo could you please assist here?

[looooo]

It is a bit difficult. Installation with pip should work. But for the appimage it's a bit more difficult. The appimage doesn't use system libs and therefor you have to install pyrate locally (--user). The other option is to install pyrate via the freecad addon-manager. But I haven't tested this yet, so I am not sure if it works.

In theory adding the pyrate-repo to ~/.FreeCAD/Mod should add ~/.FreeCAD/Mod/pyrate to the sys.path (old way to load libraries) and after this ~/FreeCAD/Mod/pyrate/freecad/PyrateWorkbench should be a extension module for freecad (so you should have access via from freecad import PyrateWorbench) and pyrate itself should be directly accessible import pyrateoptics.

For installing pyrate in developer mode, it's best to use pip install -e .. But I am not sure if this also works for the appimage (pip install -e . --user)

[joha2]

The main problem with the AppImage is that I can install pyrate by installing it in the Ext or Mod directory as you already mentioned, but Pyrate is not able to find scipy if its only installed via pip --user and not system wide. Thats a bit strange. I don't know how the AppImage searches for Python libraries. Do you have an idea?

[looooo]

The appimage doesn't use system python libraries. This is by design to avoid binary incompatibilities.

What we can do (and I think it's a good idea) is to add a compatible scipy to the appimage. It's a very usefuly library also for other workbenches...

Are there any other libraries you need for pyrate?

[joha2]

Sounds like a good plan! Pyrate uses:

numpy, scipy, sympy, matplotlib, ruamel.yaml

All other stuff should be within the standard library. Could you implement them into the AppImage?

Best wishes Johannes

[looooo]

Hmm, We already include pyyaml. I guess we shouldn't include 2 yaml libraries. Matplotlib and numpy should be included. So we miss scipy and sympy. Maybe you can start a discussion on the freecad forum to see what others think about it. The problem is the size of the appimage which for sure will increase.

[joha2]

As far as I know pyyaml is not further developed. Therefore maybe it would be good to change to ruamel.yaml since it is more recent and has an implementation of yaml 1.2. I will discuss this at the thread in the forum, you suggested. I think scipy and sympy are not that large, but as far as I know scipy was never included into FreeCAD, maybe due to size limitations. I don't know.

[looooo]

I guess @easyw was the one asking for pyyaml. Also bolts uses pyyaml. So I guess these have to be ported to ruamel.yaml...

[easyw]

Pyyaml seems to be still developed... 2018-07-05: PyYAML 3.13 is released. PyYAML requires Python 2.7 or Python 3.4+.

[looooo]

thanks @easyw ruamel.yaml is slightly bigger in size.

@joha2 is ruamel.yaml really necessary? I see this has changed recently: https://github.com/joha2/pyrate/commit/a989f69a9cc7c5d08c95f7428ee64a2d2bbbeda6

[joha2]

@looooo and @easyw thanks for your comments. I discussed a refactoring of pyrate with the developer of ruamel.yaml, recently. In fact I thought ruamel is outdated which he showed me it is not. And since I thought: having a direct wire is not bad, we changed to ruamel.yaml. For us, it is not absolutely necessary and since I do not know any features of yaml 1.2, we absolutely need, we can easily switch back. I will do this soon after verifying, that a new switch does not break anything. How is the plan for the other libs?

[looooo]

I like sympy and scipy and think that these libraries are useful for freecad. scipy is compressed about 18mb. Which is about half the size of the compressed freecad-library (without docs) sympy is about 10mb. I guess it's worth to add them. Hopefully they will not add too much dependencies. I give it a try.

[looooo]

size will increase about 25%. But I think there is no other way.

[looooo]

getting up to 350 mb for these deps:

>>> import yaml
>>> import numpy
>>> import scipy
>>> import pandas
>>> import sympy
>>> import matplotlib

[sgrogan]

The intent of the AppImage is to provide a package that is distro agnostic and conflict free. The problem is that the package gets huge quickly. Luckily @triplus has implemented the AppImage.zsync

So only deltas will be downloaded by the user on an update. This offers the ability to add lots of stuff for the user. I believe the only limitation is 2 GB for the main package. We may stress the CI's some for the package upload, but I think it will be OK for the users downloads.

It's also possible to unpack the AppImage and add stuff and re-pack. This is not friendly and the user must be sure to add compatible packages.

[joha2]

Another possibility is to provide two AppImages. Since yaml, numpy, scipy, pandas, sympy and matplotlib are the most used libraries for scientific computing (together with some other smaller libraries -- maybe a thread in the forum would help), it makes total sense to provide them in an AppImage. The user should decide whether to use an AppImage with mostly no scientific libraries (if he or she only uses "standard" functionality of FreeCAD) or with these scientific libraries implemented, which leads to a larger package (that's the price we have to pay), but on the other hand makes more sophisticated scientific computing available.

[looooo]

So I guess it should be ok to include all the libraries needed for pyrate: https://github.com/FreeCAD/FreeCAD-AppImage/pull/13/commits/a7978b81969069fda0c137d6451135763c2246cc

Creating one appimage with and one appimage without these libraries doesn't make much sense to me. This will only lead to question why appimage A works while appimage B doesn't work...

Anyone who want a more customized bundle should use a package-manager.

[joha2]

Thanks @looooo What do you recommend should be added to the README.md to be able to use the appimage together with pyrateoptics?

[looooo]

I guess we need some testing before adding suggestions on how to install. But theoretically the appimage (0.18 release) should include all needed dependencies and therefor the addon-manager should work fine.

[joha2]

@looooo I reverted the ruamel.yaml dependency to pyyaml. Now everything in the app image should be fine.

[joha2]

@looooo Sorry for reopening this old issue: I realized (after updating setup.py) that only "freecad/Pyrateworkbench" is installed into the site-packages directory of my environment. Is this sufficient without all the ui files for the dialogs? Further at the moment the ui files get loaded via

fn = getRelativeFilePath(__file__, 'Qt/dlg_field.ui')

Shouldn't that be via the usual Python package data access methods? What do you think?

I have the feeling that at the moment via the AddonManager there is some kind of "doppelte Buchfuehrung" due to using the clone github directory for accessing the dialog data files and using the sitepackages directory for accessing the Python files via import freecad.... and so on. Is this correct?

Best wishes Johannes

[looooo]

I realized (after updating setup.py) that only "freecad/Pyrateworkbench" is installed into the site-packages directory of my environment. Is this sufficient without all the ui files for the dialogs?

you can add all the additional files to the MANIFEST.in.

Shouldn't that be via the usual Python package data access methods? What do you think?

Not sure what the usual Python package acess methods are. Also not sure what the getRelativeFilePath method does, but if it does something like getting the directory-path of file and adding the second argument, I think this should be just fine.

I have the feeling that at the moment via the AddonManager there is some kind of "doppelte Buchfuehrung" due to using the clone github directory for accessing the dialog data files and using the sitepackages directory for accessing the Python files via import freecad.... and so on. Is this correct?

Yes, that is still a problem. The addon-manager lacks dependency handling and pip lacks easy usage. As I said in the private post already, we can try to promote pyrate a little bit by including it in the bundles directly.

[joha2]

Hey @looooo thanks for your reply. The getRelativeFilePath method does exactly what you think. By usual package access method, I mean the methods from https://setuptools.readthedocs.io/en/latest/pkg_resources.html#resourcemanager-api to manage access to resources within a package.

Best regards Johannes