newbie questions - Githubissues

omarcostahamido commented 3 years ago

Hi @shakfu What a great repo! I have some newbie questions if you don't mind me asking:

is this mac os only?
does it only work with homebrew python installation or vanilla python from python.org also works?
does it actually embeds python into the external or is the external connecting to local python installation?
could you add a simple list of dependencies to the readme?
any reason why you can't provide already compiled max externals?
- I know you probably don't want to populate the repo history with it but you could always add them to a release?
does this work with python packages?
- is it possible to provide an example that installs for example numpy and makes use of it?

Thank you for your attention and sorry for the long list of questions. Best, OCH

omarcostahamido commented 2 years ago

Right... the problem is that if I just remove missing_ok it will throw an error of file missing and stop the process:

$  python3 -m builder pyjs_shared_pkg -i -b
30   INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 165, in <module>
    app.cmdline()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 110, in cmdline
    options.func(self, options)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 139, in do_pyjs_shared_pkg
    self.ordered_dispatch(config.shared_pkg_builder, args)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 67, in ordered_dispatch
    getattr(builder, key)()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1047, in install
    builder.install()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 754, in install
    self.reset()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 377, in reset
    self.cmd.remove(self.src_path)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 166, in remove
    path.unlink()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1287, in unlink
    self._accessor.unlink(self)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3'

needs a clever workaround here...

shakfu commented 2 years ago

Hi @omarcostahamido

That's a strange error. The remove function on line 159 is called by reset on line 377 at the beginning of the build process to remove the old src directory.

    def reset(self):
        """remove product src directory and compiled product directory."""
        #     for builder in self.depends_on:
        #         builder.reset()
        self.cmd.remove(self.src_path)
        self.cmd.remove(self.prefix)

and

    def remove(self, path):
        """Remove file or folder."""
        if path.is_dir():
            self.log.info("remove folder: %s", path)
            shutil.rmtree(path, ignore_errors=(not DEBUG))
        else:
            self.log.info("remove file: %s", path)
            path.unlink(missing_ok=True)

missing_ok only kicks in for a file. Did you change the DEBUG setting by any chance?

omarcostahamido commented 2 years ago

Didn't change DEBUG setting (not sure how to). My fork is up to date with you master branch. https://github.com/omarcostahamido/py-js

O

On Wed, Mar 16, 2022 at 1:33 PM Shakeeb Alireza @.***> wrote:

Hi @omarcostahamido https://github.com/omarcostahamido

That's a strange error. The remove function on line 159 is called by reset on line 377 at the beginning of the build process to remove the old src directory.
def reset(self):
    """remove product src directory and compiled product directory."""
    #     for builder in self.depends_on:
    #         builder.reset()
    self.cmd.remove(self.src_path)
    self.cmd.remove(self.prefix)
and
def remove(self, path):
    """Remove file or folder."""
    if path.is_dir():
        self.log.info("remove folder: %s", path)
        shutil.rmtree(path, ignore_errors=(not DEBUG))
    else:
        self.log.info("remove file: %s", path)
        path.unlink(missing_ok=True)
missing_ok only kicks in for a file. Did you change the DEBUG setting by any chance?

— Reply to this email directly, view it on GitHub https://github.com/shakfu/py-js/issues/2#issuecomment-1069134533, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEL4NAAPIZFE2NHECOXOZHDVAHPJJANCNFSM47KVWMXQ . You are receiving this because you were mentioned.Message ID: @.***>

shakfu commented 2 years ago

When you ran python3 -m builder pyjs_shared_pkg -i -b did it download a bunch of things (python dependencies and python3 src code itself from python.org?

Can you please try another experiment with some code I just pushed. At the root of your py-js fork which after you update it, can you do the following:

./build.sh

It should pick up your system python3 and create externals in externals. It should not be homebrew specific any longer..

shakfu commented 2 years ago

I ran python3 -m builder pyjs_shared_pkg -i -b on a clean clone and I didn't have any issues.

Check out the py-js/source/py/targets/build directory. It should look some like this after you run the above:

.
├── downloads
│   ├── Python-3.9.10.tgz
│   ├── bzip2-1.0.8.tgz
│   ├── openssl-1.1.1g.tgz
│   └── xz-5.2.5.tgz
├── lib
│   ├── bzip2
│   ├── openssl
│   ├── python-shared
│   └── xz
└── src
    ├── Python-3.9.10
    ├── bzip2-1.0.8
    ├── openssl-1.1.1g
    └── xz-5.2.5

11 directories, 4 files

shakfu commented 2 years ago

Ok, I appreciate you are having issues due to your use of python3.7

I bit the bucket and downloaded it and installed it. I just pushed a fix on the default sys build. From root of py-js, just run the following now:

./build.sh

I tested it on python3.7.8 and it builds fine.

I'll try python3 -m builder pyjs_shared_pkg -i -b as well with 3.7.8

Let's see.

omarcostahamido commented 2 years ago

Hey @shakfu

I bit the bucket ...

That's a new expression I learned today. I knew to bite the bullet... maybe that's what you meant? haha 😆

Running ./build.sh at the root just builds the two minimal externals successfully! And yes, on my machine with Python 3.7.3 from python.org 🎉 *however this doesn't seem to be downloading the python source to the support folder (as it should?).

Now... how do I use pip in here?...

Running python3 -m builder pyjs_shared_pkg -i -b inside source/py still throws TypeError: unlink() got an unexpected keyword argument 'missing_ok'

shakfu commented 2 years ago

Hi @omarcostahamido

Great. Looks like the default ./build.sh worked for you. These externals are 'local', linked to your system site-packages so you can pretty much install anything there as normal via pip or otherwise, and they'll be available to Max via the these two externals. There is no downloading whatsoever. They just work (-:

The tradeoff is that they just work for you and can't be shared with others.

For the latter, I fixed the python 3.7 issues in the code-base, so now running:

cd py-js/source/py
python3 -m builder pyjs_shared_pkg -i -b

should just work (I only recently pushed my commit) as expected.

Let me know if doesn't.

Thanks!

omarcostahamido commented 2 years ago

Hi @shakfu

Got it, the ./build.sh version is still the same version that is connected to my system python installation. ✔️

I updated my for ran the command and... It definitely started downloading something this time around. But it failed at some point. I know this is a lot of lines but I want you to be able to see exactly what's going on. So here it goes:

all the output from the console

``` $ python3 -m builder pyjs_shared_pkg -i -b 32 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3 32 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3 32 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared 32 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared 33 INFO : Bzip2Builder downloading https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz 33 INFO : Bzip2Builder curl -L --fail https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 791k 100 791k 0 0 569k 0 0:00:01 0:00:01 --:--:-- 569k curl: (7) Couldn't connect to server 1438 INFO : Bzip2Builder unpacking /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/bzip2-1.0.8 1438 INFO : Bzip2Builder tar -C /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src -xvf /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz tar: Must specify one of -c, -r, -t, -u, -x 1450 INFO : OpensslBuilder downloading https:/www.openssl.org/source/openssl-1.1.1g.tar.gz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/openssl-1.1.1g.tgz 1450 INFO : OpensslBuilder curl -L --fail https:/www.openssl.org/source/openssl-1.1.1g.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/openssl-1.1.1g.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9571k 100 9571k 0 0 13.5M 0 --:--:-- --:--:-- --:--:-- 13.5M curl: (7) Couldn't connect to server 2152 INFO : OpensslBuilder unpacking /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/openssl-1.1.1g 2152 INFO : OpensslBuilder tar -C /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src -xvf /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/openssl-1.1.1g.tgz tar: Must specify one of -c, -r, -t, -u, -x 2157 INFO : XzBuilder downloading http:/tukaani.org/xz/xz-5.2.5.tar.gz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/xz-5.2.5.tgz 2157 INFO : XzBuilder curl -L --fail http:/tukaani.org/xz/xz-5.2.5.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/xz-5.2.5.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 246 100 246 0 0 1118 0 --:--:-- --:--:-- --:--:-- 1113 100 257 100 257 0 0 507 0 --:--:-- --:--:-- --:--:-- 507 100 964 0 964 0 0 1039 0 --:--:-- --:--:-- --:--:-- 1039 100 321 100 321 0 0 196 0 0:00:01 0:00:01 --:--:-- 583 100 1749k 100 1749k 0 0 520k 0 0:00:03 0:00:03 --:--:-- 1120k curl: (7) Couldn't connect to server 5534 INFO : XzBuilder unpacking /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/xz-5.2.5 5534 INFO : XzBuilder tar -C /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src -xvf /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/xz-5.2.5.tgz tar: Must specify one of -c, -r, -t, -u, -x 5540 INFO : SharedPythonForPkgBuilder downloading https:/www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/Python-3.7.3.tgz 5540 INFO : SharedPythonForPkgBuilder curl -L --fail https:/www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/Python-3.7.3.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 21.9M 100 21.9M 0 0 23.2M 0 --:--:-- --:--:-- --:--:-- 23.1M curl: (7) Couldn't connect to server 6494 INFO : SharedPythonForPkgBuilder unpacking /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3 6495 INFO : SharedPythonForPkgBuilder tar -C /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src -xvf /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/Python-3.7.3.tgz tar: Must specify one of -c, -r, -t, -u, -x 6499 INFO : SharedPythonForPkgBuilder changing working dir to: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3 Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 165, in app.cmdline() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 110, in cmdline options.func(self, options) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 139, in do_pyjs_shared_pkg self.ordered_dispatch(config.shared_pkg_builder, args) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 67, in ordered_dispatch getattr(builder, key)() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1080, in install builder.install() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 920, in install self.pre_process() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 911, in pre_process self.cmd.chdir(self.src_path) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 140, in chdir os.chdir(path) FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3' ```

shakfu commented 2 years ago

Hi @omarcostahamido

The errors you are showing are download (network-related) errors. I can't much help you these. Perhaps try later?

omarcostahamido commented 2 years ago

Hi @shakfu

It is pretty much consistent:

(...)
51   INFO : Bzip2Builder curl -L --fail https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  791k  100  791k    0     0   542k      0  0:00:01  0:00:01 --:--:--  542k
curl: (7) Couldn't connect to server
(...)
1529 INFO : OpensslBuilder curl -L --fail https:/www.openssl.org/source/openssl-1.1.1g.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/openssl-1.1.1g.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9571k  100 9571k    0     0  1254k      0  0:00:07  0:00:07 --:--:-- 1328k
curl: (7) Couldn't connect to server
(...)
9174 INFO : XzBuilder  curl -L --fail http:/tukaani.org/xz/xz-5.2.5.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/xz-5.2.5.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   246  100   246    0     0   1083      0 --:--:-- --:--:-- --:--:--  1083
100   257  100   257    0     0    317      0 --:--:-- --:--:-- --:--:-- 32125
100   968    0   968    0     0    652      0 --:--:--  0:00:01 --:--:--   652
100   325  100   325    0     0    121      0  0:00:02  0:00:02 --:--:--   442
100 1749k  100 1749k    0     0   366k      0  0:00:04  0:00:04 --:--:--  846k
curl: (7) Couldn't connect to server
(...)
13961 INFO : SharedPythonForPkgBuilder curl -L --fail https:/www.python.org/ftp/python/3.7.3/Python-3.7.3.tgz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/Python-3.7.3.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21.9M  100 21.9M    0     0  1001k      0  0:00:22  0:00:22 --:--:-- 1489k
curl: (7) Couldn't connect to server
(...)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3'

I wonder what else it is trying to download that is failing?

shakfu commented 2 years ago

@omarcostahamido

The problem is that you are trying to download a version of python which is no longer available. (see https://python.org). The latest (and only version) of 3.7 downloadable is 3.7.8 and you are seemingly try to download 3.7.3

While I'd love to support older versions of Python, version 3.7 is particularly irritating because of the aberrational and inconsistent use of the suffix 'm' in python3.7m and libpython3.7m

This means that I have to write a whole bunch of code just to address that irritating special case, which is way too much time and effort. So I'm not going to spend any more time on 3.7 compatibility. If its very important for you please feel free to suggest some PRs, otherwise, i suggest you update to 3.8 or 3.9 or 3.10

shakfu commented 2 years ago

Hi @omarcostahamido

After some further thought about supporting more python versions, I did some further research and it seems that one can sidestep special-casing the differences between python versions to an extent and automatically configure the compilation variables. The key python module, which I confess I didn't ever use before, is called sysconfig and it basically makes my task a lot lot easier.

Here's a short demo of what I mean. Create the following python script, let's call it py_config.py:

import sysconfig

VARS = [
    'py_version',
    'py_version_short',
    'py_version_nodot',
    'abiflags',
    'prefix',
    'BINDIR',
    'INCLUDEPY',
    'LIBDIR',
    'LIBRARY',
    'LDLIBRARY'
]

for v in VARS:
    print(v, ':', sysconfig.get_config_var(v))

If it is run by your conventionally installed python3.7 from python.org

$ python3 py_config.py
py_version : 3.7.8
py_version_short : 3.7
py_version_nodot : 37
abiflags : m
prefix : /Library/Frameworks/Python.framework/Versions/3.7
BINDIR : /Library/Frameworks/Python.framework/Versions/3.7/bin
INCLUDEPY : /Library/Frameworks/Python.framework/Versions/3.7/include/python3.7m
LIBDIR : /Library/Frameworks/Python.framework/Versions/3.7/lib
LIBRARY : libpython3.7m.a
LDLIBRARY : Python.framework/Versions/3.7/Python

If run using my Homebrew install python3.9:

$ /usr/local/bin/python3 py_config.py
py_version : 3.9.10
py_version_short : 3.9
py_version_nodot : 39
abiflags :
prefix : /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9
BINDIR : /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/bin
INCLUDEPY : /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9
LIBDIR : /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib
LIBRARY : libpython3.9.a
LDLIBRARY : Python.framework/Versions/3.9/Python

Pretty cool. After 20 years of python, I am still being pleasantly surprised :smile:

omarcostahamido commented 2 years ago

Dear @shakfu

I'm happy my questioning led you to discover sysconfig. Indeed it makes sense that such a thing exists, and that you don't have to hard code it every time. But I am really glad that it just makes it completely optional the way you install python now :) I understand you don't want to keep wasting time trying to make your project compatible with every single version of Python, and again, thank you so much for engaging with me through all this. However, if you still allow me, let me ask you one more thing related to this:

The problem is that you are trying to download a version of python which is no longer available. (see https://python.org/). The latest (and only version) of 3.7 downloadable is 3.7.8 and you are seemingly try to download 3.7.3

I note that the latest python version available is 3.7.13, not 3.7.8, and that the 3.7.3 version, including the source code, is indeed available here https://www.python.org/downloads/source/ I just haven't found the place or the reason why your script can't access it.

But, from a totally different perspective I was also wondering... why should it be trying to download the same exact version as the one installed on the system? If this script is downloading the source code for python and compiling it from scratch to create an entirely independent max object (that will not be connected to the system python installation) why should it even matter if it is downloading the same exact version installed on the system? (it might even be beneficial to be able to choose different versions to compile).

Thank you for sharing your thoughts.

omarcostahamido commented 2 years ago

Also, I just tried on Python 3.8.9 and I am facing the same issues:

console output

``` $ python3 -m builder pyjs_shared_pkg -i -b 25 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.8.9 25 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.8.9 26 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared 26 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared 26 INFO : Bzip2Builder downloading https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz 26 INFO : Bzip2Builder curl -L --fail https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 791k 100 791k 0 0 391k 0 0:00:02 0:00:02 --:--:-- 391k curl: (7) Couldn't connect to server 2061 INFO : Bzip2Builder unpacking /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/bzip2-1.0.8 2061 INFO : Bzip2Builder tar -C /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src -xvf /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz tar: Must specify one of -c, -r, -t, -u, -x 2070 INFO : OpensslBuilder downloading https:/www.openssl.org/source/openssl-1.1.1g.tar.gz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/openssl-1.1.1g.tgz 2070 INFO : OpensslBuilder curl -L --fail https:/www.openssl.org/source/openssl-1.1.1g.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/openssl-1.1.1g.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 9571k 100 9571k 0 0 2534k 0 0:00:03 0:00:03 --:--:-- 2539k curl: (7) Couldn't connect to server 5861 INFO : OpensslBuilder unpacking /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/openssl-1.1.1g 5861 INFO : OpensslBuilder tar -C /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src -xvf /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/openssl-1.1.1g.tgz tar: Must specify one of -c, -r, -t, -u, -x 5870 INFO : XzBuilder downloading http:/tukaani.org/xz/xz-5.2.5.tar.gz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/xz-5.2.5.tgz 5870 INFO : XzBuilder curl -L --fail http:/tukaani.org/xz/xz-5.2.5.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/xz-5.2.5.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 246 100 246 0 0 190 0 0:00:01 0:00:01 --:--:-- 191 100 257 100 257 0 0 145 0 0:00:01 0:00:01 --:--:-- 934 100 964 0 964 0 0 355 0 --:--:-- 0:00:02 --:--:-- 0 100 321 100 321 0 0 85 0 0:00:03 0:00:03 --:--:-- 966 100 1749k 100 1749k 0 0 60798 0 0:00:29 0:00:29 --:--:-- 96286 curl: (7) Couldn't connect to server 35353 INFO : XzBuilder unpacking /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/xz-5.2.5 35353 INFO : XzBuilder tar -C /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src -xvf /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/xz-5.2.5.tgz tar: Must specify one of -c, -r, -t, -u, -x 35359 INFO : SharedPythonForPkgBuilder downloading https:/www.python.org/ftp/python/3.8.9/Python-3.8.9.tgz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/Python-3.8.9.tgz 35359 INFO : SharedPythonForPkgBuilder curl -L --fail https:/www.python.org/ftp/python/3.8.9/Python-3.8.9.tgz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/Python-3.8.9.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 23.3M 100 23.3M 0 0 2950k 0 0:00:08 0:00:08 --:--:-- 2890k curl: (7) Couldn't connect to server 43479 INFO : SharedPythonForPkgBuilder unpacking /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.8.9 43479 INFO : SharedPythonForPkgBuilder tar -C /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src -xvf /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/Python-3.8.9.tgz tar: Must specify one of -c, -r, -t, -u, -x 43485 INFO : SharedPythonForPkgBuilder changing working dir to: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.8.9 Traceback (most recent call last): File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 165, in app.cmdline() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 110, in cmdline options.func(self, options) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 139, in do_pyjs_shared_pkg self.ordered_dispatch(config.shared_pkg_builder, args) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 67, in ordered_dispatch getattr(builder, key)() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1144, in install builder.install() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 984, in install self.pre_process() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 975, in pre_process self.cmd.chdir(self.src_path) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 195, in chdir os.chdir(path) FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.8.9' ```

omarcostahamido commented 2 years ago

Ok, I just realized your script is creating a weird file on my documents folder:

Perhaps what is going on has to do with the fact that the directory "Max 8" has a space in it?

O

shakfu commented 2 years ago

Thanks @omarcostahamido

Definitely appreciate your feedback which is helping me to improve this project.

With respect to your questions:

I note that the latest python version available is 3.7.13, not 3.7.8, and that the 3.7.3 version, including the source code, is indeed available here https://www.python.org/downloads/source/

You are right. My mistake. There's a 3.7.13 src release but the last binary release was 3.7.8.

I just haven't found the place or the reason why your script can't access it.

Looking at your errors, I don't think it's a problem with my script.

Can you try the following from your terminal:

$ curl -L --fail https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz -o bzip2.tgz

and then this

$ curl -L --fail https:/www.openssl.org/source/openssl-1.1.1g.tar.gz -o openssl.tgz

and this

curl -L --fail http:/tukaani.org/xz/xz-5.2.5.tar.gz -o xz.tgz

and finally this

curl -L --fail https:/www.python.org/ftp/python/3.8.9/Python-3.8.9.tgz -o python.tgz

omarcostahamido commented 2 years ago

Can you try the following from your terminal: (...)

Sure thing. And here is the output:

$  curl -L --fail https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz -o bzip2.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  791k  100  791k    0     0   597k      0  0:00:01  0:00:01 --:--:--  597k

$  curl -L --fail https:/www.openssl.org/source/openssl-1.1.1g.tar.gz -o openssl.tgz  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 9571k  100 9571k    0     0   388k      0  0:00:24  0:00:24 --:--:--  137k

$  curl -L --fail http:/tukaani.org/xz/xz-5.2.5.tar.gz -o xz.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   246  100   246    0     0    980      0 --:--:-- --:--:-- --:--:--   976
100   257  100   257    0     0    320      0 --:--:-- --:--:-- --:--:--   320
100   964    0   964    0     0    668      0 --:--:--  0:00:01 --:--:--  2894
100   321  100   321    0     0    122      0  0:00:02  0:00:02 --:--:--   798
100 1749k  100 1749k    0     0  45678      0  0:00:39  0:00:39 --:--:-- 90933

$  curl -L --fail https:/www.python.org/ftp/python/3.8.9/Python-3.8.9.tgz -o python.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 23.3M  100 23.3M    0     0  2870k      0  0:00:08  0:00:08 --:--:-- 2735k

(P.S. did you see my previous message here?)

shakfu commented 2 years ago

Ok, I just realized your script is creating a weird file on my documents folder:

That's weird.

In the current codebase, I don't touch HOME$/Documents/Max 8/Packages except to create symlink between the py-js repo (where you cloned the repo) and the Max Packages folder so that it becomes HOME$/Documents/Max 8/Packages/py-js, however earlier incarnations created a package called py there with the new externals.

Perhaps it's residual of that.

shakfu commented 2 years ago

Hi @omarcostahamido

Sure thing. And here is the output:

Ok then curl is working fine. It may be the tar command.

In your version of tar, does it have the -C option? From the error earlier it doesn't look like it does. That may be the problem.

omarcostahamido commented 2 years ago

In your version of tar, does it have the -C option? From the error earlier it doesn't look like it does. That may be the problem.

Hum... info tar shows this


-c      Create a new archive containing the specified items.  The long

             option form is --create.

Is that is?

On Thu, Mar 17, 2022 at 1:44 PM Shakeeb Alireza @.***> wrote:

Hi @omarcostahamido https://github.com/omarcostahamido

Sure thing. And here is the output:

Ok then curl is working fine. I may be the tar command.

In your version of tar, does it have the -C option? From the error earlier it doesn't look like it does. That may be the problem.

— Reply to this email directly, view it on GitHub https://github.com/shakfu/py-js/issues/2#issuecomment-1070942437, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEL4NADHHPE5UEGHJJHSAALVAMZLLANCNFSM47KVWMXQ . You are receiving this because you were mentioned.Message ID: @.***>

shakfu commented 2 years ago

I just pushed a commit which changes the tar command slightly (but equivalently):

tar -C <output_directory> xvf <archive>

# to

tar -xvf <archive> --directory  <output_directory>

I've also put a bunch of asserts in the preliminary steps in case any of them fail, to force a stop.

Can you please try again with this change.

shakfu commented 2 years ago

Actually scratch that. I am have some server issues again and I can't push any commits right now.

Back to GitHub support.

omarcostahamido commented 2 years ago

I am having github issues too... I think most ppl are right now. look: https://www.githubstatus.com/

On Thu, Mar 17, 2022 at 2:16 PM Shakeeb Alireza @.***> wrote:

Actually scratch that. I am have some server issues again and I can't push any commits right now.

Back to GitHub support.

— Reply to this email directly, view it on GitHub https://github.com/shakfu/py-js/issues/2#issuecomment-1070961204, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEL4NAAFJNFMCAADJAC3DHTVAM5FJANCNFSM47KVWMXQ . You are receiving this because you were mentioned.Message ID: @.***>

omarcostahamido commented 2 years ago

Sorry @shakfu I only seen this previous message now.

I updated my fork and tried it and... it fails faster haha:

console output

``` $ python3 -m builder pyjs_shared_pkg -i -b 34 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3 34 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3 34 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared 34 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared 34 INFO : Bzip2Builder downloading https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz 34 INFO : Bzip2Builder curl -L --fail https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 791k 100 791k 0 0 583k 0 0:00:01 0:00:01 --:--:-- 583k curl: (7) Couldn't connect to server Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 165, in app.cmdline() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 110, in cmdline options.func(self, options) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 139, in do_pyjs_shared_pkg self.ordered_dispatch(config.shared_pkg_builder, args) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 67, in ordered_dispatch getattr(builder, key)() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1127, in install builder.install() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 966, in install self.download() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 452, in download dep.download() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 458, in download assert self.download_path.exists(), f"could not download: {self.download_path}" AssertionError: could not download: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tgz ```

shakfu commented 2 years ago

Your download issues are a mystery. I've searched online for the curl error: "curl: (7) Couldn't connect to server" and it could be related to a proxy, firewall or https vs http availability.

Can you please try the following script, and run it some tmp directory? It's pretty straightforward, and will create two folders, download and src, and download to the former and then extract to the latter.

It's very close to how it's done on builder, and it should localize the issue.

"""
This is pretty replicates how downloading and uncompressing
is down on `builder`.

If some of the the urls below give the same error
as before 'curl: (7) Couldn't connect to server'

Then change 'https' to 'http' and rerun.

"""

import os
from pathlib import Path

URLS = [
    'https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz',
    'https:/www.openssl.org/source/openssl-1.1.1g.tar.gz',
    'http:/tukaani.org/xz/xz-5.2.5.tar.gz',
    'https:/www.python.org/ftp/python/3.8.9/Python-3.8.9.tgz',
]

def download(url):
    """download src using curl and tar.

    curl and tar are automatically available on mac platforms.
    """
    download_path = Path('downloads')
    src_path = Path('src')
    url = Path(url)

    # download
    if not download_path.exists():
        download_path.mkdir()
    dst = download_path / url.name
    if not dst.exists():
        print(f"downloading {url} to {dst}")
        os.system(f"curl -L --fail {url} -o {dst}")

    src = dst
    dst = src_path / url.stem.rstrip('.tar')

    # unpack
    if not src_path.exists():
        src_path.mkdir(parents=True, exist_ok=True)
    if not dst.exists():
        print(f"unpacking .. {src} -> {dst}")
        os.system(f"tar -xvf {src} --directory {src_path}")

for url in URLS:
    download(url)

omarcostahamido commented 2 years ago

Hi @shakfu

This script you wrote seems to work fine, at least I didn't find complains in the very long and fast log it printed. long console log.txt

shakfu commented 2 years ago

Hi @omarcostahamido

This script you wrote seems to work fine, , at least I didn't find complains in the very long and fast log it printed.

Ok, That's good. I'm assuming then that the current method in builder doesn't work for you.

shakfu commented 2 years ago

@omarcostahamido

Ok I pushed a change. Let's see if this works for you.

omarcostahamido commented 2 years ago

@shakfu

Thank you for the update. It fails fast, it seems like it didn't resolve the proper link?

console output

``` $ python3 -m builder pyjs_shared_pkg -i -b 33 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3 33 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/src/Python-3.7.3 33 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared 33 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared 33 INFO : Bzip2Builder downloading https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz to /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tar.gz 33 INFO : Bzip2Builder curl -L --fail https:/sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz -o /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 791k 100 791k 0 0 580k 0 0:00:01 0:00:01 --:--:-- 580k curl: (7) Couldn't connect to server Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 167, in app.cmdline() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 110, in cmdline options.func(self, options) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 141, in do_pyjs_shared_pkg self.ordered_dispatch(config.shared_pkg_builder, args) File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 68, in ordered_dispatch getattr(builder, key)() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1119, in install builder.install() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 960, in install self.download() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 447, in download dep.download() File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 453, in download assert self.download_path.exists(), f"could not download: {self.download_path}" AssertionError: could not download: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/downloads/bzip2-1.0.8.tar.gz ```

and (as I mentioned before) it also keeps creating this weird Max (no extension) file in the Documents folder...

shakfu commented 2 years ago

Hi @omarcostahamido

That's strange. I'm running python 3.7.8 now as my main python to figure out all of the issues for 3.7 users, and I haven't had the type of curl download error you are experiencing. Also I'm quite interested in the strange file popping up in your Documents folder.

I've scanned all of the code of builder and there's nothing that writes to the Documents folder except a symlink. Do you know what is the content of this file is it binary, text?

Are you running py-js inside your Documents/Max 8/Packages folder directly. If so this may be the issue.

Download the py-js repo somewhere where there is no iCloud activity. I download mine into $HOME/Downloads and then symlink to Document/Max 8/Packages/py-js. If you don't do this, iCloud interferes with the source code and it can cause strange errors.

I'm sorry I can't be more helpful, my current code is in flux as I'm rewriting the command line configuration code so that once can specify arbitrary versions of python to download and build into externals.

S

omarcostahamido commented 2 years ago

Hi @shakfu

Are you running py-js inside your Documents/Max 8/Packages folder directly.

Yes, I am.

At this point I am pretty sure the issue arrises when the path for the py-js includes a space somewhere. Note that if the string referring to the path is cut at the point where there is a space, then it ends up in something like .../Documents/Max. (the second part would be 8/Packages/py-js...). This is also consistent with what I've seen on another machine with python 3.8.9, where it created the mysterious Max file.

Question: can you try this on your end? Clone your repo to somewhere where there is a space in the path, like /Downloads/test pyjs/py-js/

I've cloned the repo to another location who's path doesn't include a space, and... it was doing a lot of things for some minutes and it ended here (complete output on the file attached bellow):

386866 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Downloads/testpyjs/py-js/source/py/targets/build/lib/python-shared/lib/python3.7/site-packages
386866 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Downloads/testpyjs/py-js/source/py/targets/build/lib/python-shared/lib/python3.7/site-packages
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Downloads/testpyjs/py-js/source/py/builder/__main__.py", line 167, in <module>
    app.cmdline()
  File "/Users/omar/Downloads/testpyjs/py-js/source/py/builder/cli.py", line 110, in cmdline
    options.func(self, options)
  File "/Users/omar/Downloads/testpyjs/py-js/source/py/builder/__main__.py", line 141, in do_pyjs_shared_pkg
    self.ordered_dispatch(config.shared_pkg_builder, args)
  File "/Users/omar/Downloads/testpyjs/py-js/source/py/builder/__main__.py", line 68, in ordered_dispatch
    getattr(builder, key)()
  File "/Users/omar/Downloads/testpyjs/py-js/source/py/builder/core.py", line 1119, in install
    builder.install()
  File "/Users/omar/Downloads/testpyjs/py-js/source/py/builder/core.py", line 963, in install
    self.post_process()
  File "/Users/omar/Downloads/testpyjs/py-js/source/py/builder/core.py", line 1011, in post_process
    self.ziplib()
  File "/Users/omar/Downloads/testpyjs/py-js/source/py/builder/core.py", line 728, in ziplib
    self.lib_dynload.rename(temp_lib_dynload)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1312, in rename
    self._accessor.rename(self, target)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/Downloads/testpyjs/py-js/source/py/targets/build/lib/python-shared/lib/python3.7/lib-dynload' -> '/Users/omar/Downloads/testpyjs/py-js/source/py/targets/build/lib/python-shared/lib/lib-dynload'

console log.txt

shakfu commented 2 years ago

Hi @omarcostahamido, thanks for your error report and patience.

If you are running py-js in $HOME/Max 8/Packages then that could explain the strange file that keeps appearing. Indeed it may be curl trying to download to a path with a space in it that's nice properly escaped. I will check my string escaping again for this.

shakfu commented 2 years ago

@omarcostahamido just pushed some additional quotations of file operations in builder... Let me know if it works for you when you get a chance. Thanks

omarcostahamido commented 2 years ago

Hi @shakfu

Thank you, that now worked similar to when I was running on a no-spaces-in-path directory. It ran for a couple minutes and then ended here:

201043 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared/lib/python3.7/site-packages
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 179, in <module>
    app.cmdline()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 110, in cmdline
    options.func(self, options)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 153, in do_pyjs_shared_pkg
    self.ordered_dispatch(config.shared_pkg_builder, args)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 80, in ordered_dispatch
    getattr(builder, key)()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1122, in install
    builder.install()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 784, in install
    self.post_process()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1013, in post_process
    self.ziplib()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 732, in ziplib
    self.lib_dynload.rename(temp_lib_dynload)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1312, in rename
    self._accessor.rename(self, target)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared/lib/python3.7/lib-dynload' -> '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared/lib/lib-dynload'

It is all really fast, but at some point I noticed these reds:

Looking forward to knowing your thoughts :)

shakfu commented 2 years ago

Hi @omarcostahamido

I suggest you pause testing pyjs_shared_pkg on python 3.7, since 3.7 is not fully working on my system yet. It compiles fine, but I've been using the patch system from 3.6 and retrofitted some patches from latter versions and I wasn't surprised these don't work. The patches are version-specific and cannot be automated, and I will have to manually go through each one and make each suitable for the particular version.

Nonetheless, pyjs_shared_pkg works without issues on 3.9 and 3.10 (I will test 3.8 after I complete 3.7).

I am also slowly switching over to fully parametrizable builds (see the newpy-js/source/py/builder/factory.py module), so it should be possible to set the desired python version for the externals from the commandline. It should also be possible to selectively switch off build minimization if you prefer a larger more full featured-externals, etc..

Finally, I've been thinking about your case, where you are cloning into the $HOME/Documents/Max 8/Packages/py-js and then building from there. While this is not recommended, I should anticipate that it could happen (as it did in your case). Therefore, there will have to be checks as to where py-js/source/py/builder is run from and to correct for this (not symlinking into $HOME/Documents/Max 8/Packages/py-js if py-js is already there is done already).

I think the biggest issue is doing a lot of builds in a directory which contains a space. One thing, I've been experimenting with is shifting the build directory from py-js/source/py/targets/build to /tmp/_pyjs_build. This is now working without issues, so may be the default in the future.

What are your thoughts on the above?

omarcostahamido commented 2 years ago

Dear @shakfu

Thank you for sharing your thoughts. I'll pause pyjs_shared_pkg and will be happy to resume when you have some updates that will need more testing. Switching over to fully parametrizable builds sounds like the way to go!

The option to switch off building minimization is interesting, but I am not exactly sure about what it means. So, I see some avenue to explore/document this: max patches that exemplify the different flavours of py-js compiled externals. (I am happy to help with this).

Finally, I've been thinking about your case, where you are cloning into the $HOME/Documents/Max 8/Packages/py-js and then building from there. While this is not recommended...

I am not sure why it is not recommended: the build process is just creating the externals, correct? So, that probably means that one would then always need to move the max patches manually? I mean, what is the purpose of structuring this repository as a Max package if it is not to be placed in the Max 8/Packages folder? My experience with other max packages with source code, including cycling '74 own's min-sdk, is that they are really meant to be placed on the packages folder and build from there. Again the only thing I've been building are max externals... I know you have a python script to programmatically build max patches too (which sounds very cool), but you're not really doing that here, right?

One thing, I've been experimenting with is shifting the build directory

That is clever workaround, yes. (just one more thing to puzzle the poor person who'll try to make this compatible with Windows, though, haha...)

These are my thoughts, and thank you for asking :)

OCH

shakfu commented 2 years ago

Hi @omarcostahamido

Thanks for your feedback and comments.

The option to switch off building minimization is interesting, but I am not exactly sure about what it means.

Sorry, I should have explained this more. By 'build minimization', I mean steps taken to shrink the footprint of the python3 external product. This entails removing things that are self-evidently not required in an external such as headers, the static python library, multi-megabytes of tests, parts of the standard library which are not relevant to an externals (like tkinter, curses, and other extensions), etc. I also remove compiled extensions for non-english codecs such as _codecs_jp, _codecs_hk, _codecs_cn. So switching it off selectively, may result in a bigger external but help in case one does need one or more of the currently removed elements.

I am not sure why it is not recommended (to use $HOME/Documents/Max 8/Packages) for py-js

Again, I should have been clearer: the recommendation is from me, and is a product of my own biases, and negative experiences building and running code inside my Documents folder. For me an external repo is a 'throwaway' construct, it can be always replaced. Whereas, the Documents folder is supposed to contain high value data, so requires much more care, and it is also (on MacOS), always replicating to the cloud -- which means changes locally can also affect cloud-based backups.

Even so, when I first started developingpy-js, I also developed in $HOME/Documents/Max 8/Packages, but then I started to get compilation errors which were a result of iCloud replication occurring at the same time as compilation, and I also lost some worked because of mistyping some command line instruction. I also go feedback early on that one my commands could possibly cause damage because it was not properly quoted. From that point onwards, I clone outside of Documents, and if required install py-js via symlink.

Nonetheless, if others will do it, I really should make building py-js in $HOME/Documents/Max 8/Packages, as safe as possible which requires ensuring that all paths are quoted, that builds are in /tmp, ensuring you don't have strange file appearing in your users' Documents folder :smile:

shakfu commented 2 years ago

Hi @omarcostahamido

Ok, the pause is over. I think you will be happier with the state of things (see the README for more details):

replaced static configuration system with new factory based configuration (builder/factory.py)
builder has more options and a new automatic two-tier command system
added new make frontend to python builder (type make help in the project root)
fixed and tested lots of build options: framework-ext, framework-pkg, and also added relocatable-pkg (check it out).
lots of small fixes..

What I haven't yet done is make the code python 3.7 compatible. or made builds default to /tmp yet. But that should be a lower hanging fruit now that the core build options are available and now that the build infrastructure is more robust.

Let me know how it goes!

shakfu commented 2 years ago

@omarcostahamido

Finishing off from my last post. Code is now tested to generate externals on python 3.7-3.10

See the updated README for further details.

This thread is become way too long, so if you are going to respond or highlight any issue, may I suggest that you start another case-specific thread.

I will take some of the learnings in this thread and then add them to a FAQ.

If there aren't major issues, I'd like to spend some time on refactoring the build code which has grown to accommodate a bunch of special cases. in any case, let me know how it works for you.

In any case, I suggest you try relocatable-pkg (see the section in the README). It pretty much does everything you asked for earlier. (-:

externals with arbitrary python versions
pip installable pkg structure
specify packages to install during build
..

omarcostahamido commented 2 years ago

Dear @shakfu

Sorry for the delay. I am reading through the readme. The make help is really helpful. Nice touch on The relocatable-python variation.

Now, I understand things have change quite a bit. But I still tried our old command which returned this:

$  python3 -m builder pyjs_shared_pkg -i -b
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 35, in <module>
    from .ext.relocatable_python import analyze, relocatablize
ImportError: cannot import name 'analyze' from 'builder.ext.relocatable_python' (unknown location)

Then, following the new readme I just tried running, from the root:

$  make shared-pkg
>>> cleaning externals
>>> cleaning support directory
>>> cleaning build lib from shared-pkg target
>>> builder pyjs shared_pkg --install --build 
/bin/sh: line 0: cd: /Users/omar/Documents/Max: No such file or directory
make: *** [shared-pkg] Error 1

On the plus side, it is good to know that it won't try to take for granted the first half of a path and start generating weird files. Please let me know which commands I should try, and an example of using the relocatable-pkg (i see the help but I am still not sure what to type and how to select a package to be included haha).

I've added some reviews to the readme on PR #7 . One thing I would also like to suggest is to add the new relocatable target to the Build Variations section.

Thank you for all your work on this!

shakfu commented 2 years ago

Hi @omarcostahamido

Thanks for the PR and the feedback as always!

The ui has changed quite a bit. The basic idea is that there's the frontend Makefile which is what you should use for default settings and then there is the python3 -m builder interface for expert mode which gives a lot more options.

If you want to use the builder interface, note that the command is now two-levels instead of just one-level:

python3 -m builder pyjs_shared_pkg -i -b

is now (dropping the underscore between pyjs and shared_pkg

python3 -m builder pyjs shared_pkg -i -b

I think the easiest thing for you to try is the following:

make relocatable-pkg

Give me a couple minutes though, I'll push some fixes now which should make things a bit more stable.

shakfu commented 2 years ago

@omarcostahamido

Sorry, please try make relocatable-pkg now. I just tested it, it works and you get pip installable packages in a Max package structure.

Let me know if you have any issues.

omarcostahamido commented 2 years ago

@shakfu

Still really not happy about a path with a space on it:

$  make relocatable-pkg
>>> cleaning externals
>>> cleaning support directory
>>> cleaning build lib from framework-pkg target
>>> builder pyjs relocatable_pkg 
/bin/sh: line 0: cd: /Users/omar/Documents/Max: No such file or directory
make: *** [relocatable-pkg] Error 1

P.S. we are 6 posts away from post no. 100!

shakfu commented 2 years ago

@omarcostahamido

Sorry about that, it was an unquoted cd ($PYDIR) in the makefile. fixed it now. Let's see how it goes

omarcostahamido commented 2 years ago

Hi @shakfu

Here is the output:

$  make relocatable-pkg
>>> cleaning externals
>>> cleaning support directory
>>> cleaning build lib from framework-pkg target
>>> builder pyjs relocatable_pkg 
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 22, in <module>
    from .ext.relocatable_python import (
ImportError: cannot import name 'process_args_for_relocatable_python' from 'builder.ext.relocatable_python' (unknown location)
make: *** [relocatable-pkg] Error 1

Dang it. But almost there!

shakfu commented 2 years ago

Hi @omarcostahamido

Thanks for providing the detailed bug report.

I just discovered what's been the source of our troubles: in the standard github c/c++ .gitignore which I have in the root of the project, there was a __* entry!!

This meant that all python __init__.py and __main__.py files were basically not committed to the repo before!! I just removed the entry, included the missing files in the repo and tried to run make relocatable-pkg from py-js in $HOME/Documents/Max 8/Packages/py-js, basically your setup and it runs fine.

What an irritating bug.

Glad it's gone now. :innocent:

Thanks for your patience and feedback, and definitely please let me know if this fix doesn't work for you.

omarcostahamido commented 2 years ago

Hi @shakfu

Thank you for detecting that. All bugs look silly once you fix them 😇

So first-time.txt

Then, I was a little bit concerned about seeing it installing python packages and decided to run it from a venv with pip and setuptools upgraded second-time.txt

And then, because the script was throwing the lazy vibes of you already have these files so I ain't downloading them again for you! xD, I decided to clean the contents of the support and remove the source/py/targets/build folder myself. Ran it again and: third-time.txt

Well, it was at least interesting to see it downloading python 3.9 this time around.

2 messages before 100. We gotta make a big celebration soon!

shakfu commented 2 years ago

Hi @omarcostahamido

Thanks for the detailed test reports.

Ok, well, at least it's clear what the issue is: relocatable-pkg defaults to downloading python 3.9.10 which works fine according to your 3rd log file.

The problems start happening because the cleaning and shrinking operations which are supposed to be done after downloading this distribution are themselves run by your system 3.7 python which thinks the downloaded Python.framework in the support folder is actually 3.7 rather than 3.9...

That's a bug in my code due to the recent desire to shrink the relocatable-pkg python in support. I was running 3.9.10 on my system and also downloading it via relocatable-pkgso the bug did not occur.

I'll fix this.

In the meantime, could you also try to run make shared-pkg or make framework-pkg

I'll be interested to see if these work for you or otherwise.

omarcostahamido commented 2 years ago

Hi @shakfu 🎉 Happy 💯 th post 🎉

I cleaned up the same places and ran make shared-pkg and it goes on for a couple minutes and ends with:

186708 INFO : SharedPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared/lib/python3.7/site-packages
186708 WARNING: SharedPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared/lib/python3.7/site-packages
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 239, in <module>
    app.cmdline()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 194, in cmdline
    options.func(self, options)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 184, in do_pyjs_shared_pkg
    self.ordered_dispatch('pyjs_shared_pkg', args)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 79, in ordered_dispatch
    getattr(builder, method)()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1392, in install
    builder.install()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 903, in install
    self.post_process()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1141, in post_process
    self.ziplib()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 851, in ziplib
    self.lib_dynload.rename(temp_lib_dynload)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1312, in rename
    self._accessor.rename(self, target)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared/lib/python3.7/lib-dynload' -> '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/python-shared/lib/lib-dynload'
make: *** [shared-pkg] Error 1

Then I just ran make framework-pkg and it is faster (probably because I didn't clean those folders?) but ended here:

17620 INFO : FrameworkPythonForPkgBuilder remove file: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/Python.framework/Versions/3.7/lib/python3.7/site-packages
17620 WARNING: FrameworkPythonForPkgBuilder file not found: /Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/Python.framework/Versions/3.7/lib/python3.7/site-packages
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 239, in <module>
    app.cmdline()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/cli.py", line 194, in cmdline
    options.func(self, options)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 194, in do_pyjs_framework_pkg
    self.ordered_dispatch('pyjs_framework_pkg', args)
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/__main__.py", line 79, in ordered_dispatch
    getattr(builder, method)()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1392, in install
    builder.install()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 903, in install
    self.post_process()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 1369, in post_process
    self.ziplib()
  File "/Users/omar/Documents/Max 8/Packages/py-js/source/py/builder/core.py", line 851, in ziplib
    self.lib_dynload.rename(temp_lib_dynload)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1312, in rename
    self._accessor.rename(self, target)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/Python.framework/Versions/3.7/lib/python3.7/lib-dynload' -> '/Users/omar/Documents/Max 8/Packages/py-js/source/py/targets/build/lib/Python.framework/Versions/3.7/lib/lib-dynload'
make: *** [framework-pkg] Error 1

Best, OCH

shakfu / py-js

newbie questions #2