JasonMendoza2008
commented
10 months ago I did sudo ln -s /usr/bin/python3 /usr/bin/python and it still doesn't work (but obviously running python in bash now works).
Open JasonMendoza2008 opened 10 months ago
JasonMendoza2008
commented
10 months ago I did sudo ln -s /usr/bin/python3 /usr/bin/python and it still doesn't work (but obviously running python in bash now works).
andraxin
commented
9 months ago @JasonMendoza2008
Apparently, the mojo binary is located at ./.modular/pkg/packages.modular.com_mojo/bin/mojo:
> find -name mojo
./.modular/.mojo_cache/mojo
./.modular/pkg/packages.modular.com_mojo/lib/mojo
./.modular/pkg/packages.modular.com_mojo/bin/mojoand trying to execute it directly reveals that the root cause has nothing to do with Python:
> ./.modular/pkg/packages.modular.com_mojo/bin/mojo
/root/.modular/pkg/packages.modular.com_mojo/bin/lldb: error while loading shared libraries: libedit.so.2: cannot open shared object file: No such file or directoryThis can be resolved by apt install -y libedit2, and voilà:
> ./.modular/pkg/packages.modular.com_mojo/bin/mojo
Welcome to Mojo! 🔥
Expressions are delimited by a blank line.
Type `:quit` to exit the REPL and `:mojo help` for further assistance.
1> print('hello clueless')
2.
hello clueless
2> BTW, please don't take offense at the word "clueless"! That's just a tip of the hat to the Modular folks who plan to take over the AI world, but cannot even write a decent installer for the only Linux distro they claim to support.
JasonMendoza2008
commented
9 months ago @andraxin thanks for the answer!
I have:
lhott@lhott:~$ find -name mojo
./.modular/pkg/packages.modular.com_mojo/lib/mojo
./.modular/pkg/packages.modular.com_mojo/bin/mojoBut when I do
lhott@lhott:~$ ./.modular/pkg/packages.modular.com_mojo/bin/mojoI get:
Illegal instruction (core dumped)I still did the install as you suggested but this didn't change anything:
lhott@lhott:~$ sudo apt install -y libedit2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libedit2 is already the newest version (3.1-20210910-1build1).
libedit2 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
lhott@lhott:~$ ./.modular/pkg/packages.modular.com_mojo/bin/mojo
Illegal instruction (core dumped)@JasonMendoza2008
Illegal instruction (core dumped)
That usually means that the binary includes an instruction your CPU doesn't support. https://duckduckgo.com/?t=ffab&q=Illegal+instruction+(core+dumped)
Chances are Mojo was compiled with optimizations that your hardware cannot cope with.
What processor are you running on? (Check: grep -m1 model\ name /proc/cpuinfo)
Sometimes, these can be silly issues, like using a bit-shift instruction like SHLX from BMI2
https://en.wikipedia.org/wiki/X86_Bit_manipulation_instruction_set#BMI2_(Bit_Manipulation_Instruction_Set_2)
instead of pure unadulterated SHL
https://en.wikibooks.org/wiki/X86_Assembly/Shift_and_Rotate
JasonMendoza2008
commented
9 months ago @andraxin I am running on: AMD Ryzen 7 5800H with Radeon Graphics
How could I fix these binaries?
andraxin
commented
9 months ago @JasonMendoza2008
How could I fix these binaries?
Given that the source code doesn't appear to be available, recompiling is not an option. You could reach out to Modular and ask what optimizations are turned on, and whether they would consider creating a more accessible executable.
Taking matters into your own hands, I'm not aware of any simple
tools for re-mapping unsupported instructions in a binary.
There's always the hard way using a disassembler to locate
the offending instructions and a hexeditor to replace them.
(If I recall correctly biew [AKA beye] does both).
Of course, you'd need to be careful to match both the size
and the functionality of any replacement.
A quick search also turns up SaBRe as potentially interesting, but I have exactly ZERO experience with that.
I guess, running in an emulator is also an option, but you might experience performance degradation when doing so.
JasonMendoza2008
commented
9 months ago andraxin
commented
9 months ago @JasonMendoza2008
If you mean a Virtual Machine, this was already what I was doing as I thought it had higher chances of working.
Running in a VM may or may not help. That would depend on what CPU architecture the VM exposes. I guess when you said you were running on AMD Ryzen 7 5800H that was referring to your host machine not the VM. Or, perhaps, your VM is set up to expose the host CPU more or less directly... If the VM emulates a legacy CPU that might actually be causing the error you reported...
JasonMendoza2008
commented
9 months ago @JasonMendoza2008
If you mean a Virtual Machine, this was already what I was doing as I thought it had higher chances of working.
Running in a VM may or may not help. That would depend on what CPU architecture the VM exposes. I guess when you said you were running on AMD Ryzen 7 5800H that was referring to your host machine not the VM. Or, perhaps, your VM is set up to expose the host CPU more or less directly... If the VM emulates a legacy CPU that might actually be causing the error you reported...
That's indeed my host machine but that's also what grep -m1 model\ name /proc/cpuinfo returns :/
andraxin
commented
9 months ago @JasonMendoza2008
That's indeed my host machine but that's also what grep -m1 model\ name /proc/cpuinfo returns :/
I cannot really accuse you of being clear... :smirk: Did you mean that on your host or inside the VM, or both? :rofl:
BTW, what host OS and VM tech are you using (if you don't mind disclosing)?
I could try to replicate. Note that after the libedit2 fix, Mojo runs fine on my box
(Intel Core i9-11900H) inside a Docker container.
JasonMendoza2008
commented
9 months ago Sorry, I was not very clear.
My host machine is running Windows 11. I'm using Oracle VM VirtualBox to install and run Ubuntu 22.04.3 LTS. I have been using it to run some other projects without any problem. So I thought I could give mojo a go within that Virtual Machine.
When I run grep -m1 model\ name /proc/cpuinfo in my Linux environment, I get AMD Ryzen 7 5800H with Radeon Graphics.
I could definitely try to set up another Virtual Machine with another environment than Ubuntu 22.04.3 LTS but I would not know which one would be the most appropriate.
andraxin
commented
9 months ago @JasonMendoza2008 I could reproduce your issue in a VirtualBox VM. I had even tried every possible paravirtualization interface option for the processor to no avail. https://superuser.com/questions/945910/how-to-select-paravirtualization-interface-in-virtualbox
Note, however, that I can run Mojo in a container on the same box (under Windows 11) just fine. I used Docker Desktop, but Rancher Desktop, or possibly even running Docker inside WSL2, could/should also work.
For a quick test, you could try the image I built: https://hub.docker.com/r/andraxin/mojo
JasonMendoza2008
commented
7 months ago @JasonMendoza2008 I could reproduce your issue in a VirtualBox VM. I had even tried every possible paravirtualization interface option for the processor to no avail. https://superuser.com/questions/945910/how-to-select-paravirtualization-interface-in-virtualbox
Note, however, that I can run Mojo in a container on the same box (under Windows 11) just fine. I used Docker Desktop, but Rancher Desktop, or possibly even running Docker inside WSL2, could/should also work.
For a quick test, you could try the image I built: https://hub.docker.com/r/andraxin/mojo
This went a bit over my head so I couldn't give it a go and I tried again today hoping it was fixed with the updates, but it still isn't (same errors)
andraxin
commented
7 months ago @JasonMendoza2008 Hmm... that's odd. I mean that you keep having the same issue with the newer version.
In the meantime, I had also tried running my original Docker image (built from 0.4.0)
on an Intel-based Mac and experienced the invalid instruction issue; BUT, rebuilding
the image using 0.7.0 fixed that. I could run the new image no problem...
On the other hand, I have not tried running in a VM on Windows since last time. I would've expected the new version to work, but I don't have hands-on experience with that combination.
Sharktheone
commented
7 months ago I have read somewhere that VirtualBox doesn't support AVX(2) with Hyper-V. Mojo needs AVX2 instructions (I somehow can't find it anymore was mentioned)
I think, VMs are (still) unsupported, but luckily Docker or WSL is. Since you are on Windows, it's probably way easier to just install Mojo in WSL.
jackos
commented
3 months ago Resolved in nightly release by https://github.com/modularml/mojo/commit/b6776222dbd1e2e72a5e6f772d01b35fee7ea0d9. Mojo now finds libpython using the python executable on the top of PATH
You can test before the official release with modular install nightly/mojo. For environments like vscode codespaces that have a python on the top of PATH without a dynamic libpython.so you can activate a virtual environment to use a different version before installation e.g. the system Python is able to link to a dynamic libpython:
/usr/bin/python3 -m venv ~/venv
source ~/venv/bin/activateOr alternatively create and activate an environment that comes with libpython like pyenv or conda
Please reopen if you have any further issues.
JasonMendoza2008
commented
3 months ago @jackos it still doesn't work. I did:
curl -s https://get.modular.com | sh -
Then
python3 -m venv mojo-venv && source mojo-venv/bin/activate
Then
modular install nightly/mojo
And I get errors, "check 42 in output failed, please submit a bug".
JasonMendoza2008
commented
3 months ago Also, it seems I cannot reopen the issue?
andraxin
commented
3 months ago @jackos it still doesn't work. I did:
curl -s https://get.modular.com | sh -Then
python3 -m venv mojo-venv && source mojo-venv/bin/activateThen
modular install nightly/mojoAnd I get errors, "check 42 in output failed, please submit a bug".
@JasonMendoza2008: I think you may be missing a crucial point in @jackos 's post.
and that is "a dynamic libpython.so". Ubuntu's "system Python" is not build that way,
as evidenced by the following (run in an Ubuntu container):
root@cbd4403178fe:/# ldd $(which python3)
linux-vdso.so.1 (0x00007fb993681000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb992fa9000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fb992f78000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fb992f5c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb992d33000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb993683000)
root@cbd4403178fe:/# That also implies that any environment created via the standard venv package
using that "system Python" will not have a dynamic libpython.so, either.
That's why @jackos suggested using pyenv or conda, since those tools build
their own Python from scratch; crucially, with a dynamic libpython.so.
BTW, the official Python container images do have a dynamic libpython.so
(in addition to their non-dynamic "system Python"); so, you could try running Mojo
in a Docker container...
$ docker run --rm -it python ldd /usr/bin/python3 /usr/local/bin/python
/usr/bin/python3:
linux-vdso.so.1 (0x00007f19dc436000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f19dc34b000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f19dc32c000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f19dc301000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f19dc120000)
/lib64/ld-linux-x86-64.so.2 (0x00007f19dc438000)
/usr/local/bin/python:
linux-vdso.so.1 (0x00007f46faea5000)
libpython3.12.so.1.0 => /usr/local/bin/../lib/libpython3.12.so.1.0 (0x00007f46fa87b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f46fa694000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f46fa5b5000)
/lib64/ld-linux-x86-64.so.2 (0x00007f46faea7000)
$ Alternatively, you could also just build your own Python with dynamic libpython.so
from scratch following the recipe embedded in the Python image's history
$ docker history --no-trunc python | grep enable-shared
<missing> 8 weeks ago RUN /bin/sh -c set -eux; wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz"; wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc"; GNUPGHOME="$(mktemp -d)"; export GNUPGHOME; gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY"; gpg --batch --verify python.tar.xz.asc python.tar.xz; gpgconf --kill all; rm -rf "$GNUPGHOME" python.tar.xz.asc; mkdir -p /usr/src/python; tar --extract --directory /usr/src/python --strip-components=1 --file python.tar.xz; rm python.tar.xz; cd /usr/src/python; gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"; ./configure --build="$gnuArch" --enable-loadable-sqlite-extensions --enable-optimizations --enable-option-checking=fatal --enable-shared --with-lto --with-system-expat --without-ensurepip ; nproc="$(nproc)"; EXTRA_CFLAGS="$(dpkg-buildflags --get CFLAGS)"; LDFLAGS="$(dpkg-buildflags --get LDFLAGS)"; make -j "$nproc" "EXTRA_CFLAGS=${EXTRA_CFLAGS:-}" "LDFLAGS=${LDFLAGS:-}" "PROFILE_TASK=${PROFILE_TASK:-}" ; rm python; make -j "$nproc" "EXTRA_CFLAGS=${EXTRA_CFLAGS:-}" "LDFLAGS=${LDFLAGS:--Wl},-rpath='\$\$ORIGIN/../lib'" "PROFILE_TASK=${PROFILE_TASK:-}" python ; make install; bin="$(readlink -ve /usr/local/bin/python3)"; dir="$(dirname "$bin")"; mkdir -p "/usr/share/gdb/auto-load/$dir"; cp -vL Tools/gdb/libpython.py "/usr/share/gdb/auto-load/$bin-gdb.py"; cd /; rm -rf /usr/src/python; find /usr/local -depth \( \( -type d -a \( -name test -o -name tests -o -name idle_test \) \) -o \( -type f -a \( -name '*.pyc' -o -name '*.pyo' -o -name 'libpython*.a' \) \) \) -exec rm -rf '{}' + ; ldconfig; python3 --version # buildkit 60.7MB buildkit.dockerfile.v0
$ BUT, it's a whole lot simpler to just use pyenv.
JasonMendoza2008
commented
3 months ago Ah I see sorry I had not understood that. I'm unfortunately too much of a beginner for this and will simply wait for a more stable version! I'm really far from being anything remotely close to decent when it comes to programming.
jackos
commented
3 months ago No worries thanks for the feedback @JasonMendoza2008. Reopening until that error message is improved. This will be in stable shortly, one catch-all way to resolve it is to run the steps here to install conda
Make sure you've run ~/miniconda3/bin/conda init --all and restarted your shell, then run:
conda create -n 3.11 python=3.11
conda activate 3.11And try reinstalling with the activated conda envionrment.
Bug description
I did:
Then, I did:
And I have the following error:
Pythonis installed (I havePython 3.10.12andPython 3.12.0).However I have to do
python3.10orpython3.12to enter Python from the bash.pythondoesn't work. Maybe that's the problem?