search

pachterlab / kb_python

A wrapper for the kallisto | bustools workflow for single-cell RNA-seq pre-processing
https://www.kallistobus.tools/
BSD 2-Clause "Simplified" License
147 stars 23 forks source link

OSError: [Errno 8] Exec format error #139

Closed akhst7 closed 3 years ago

akhst7 commented 3 years ago

Describe the issue A clear and concise description of what the issue is. I installed kb-python and run kb ref in Ubuntu focal on AWS EC2 with a following specs:

➜  ~ lscpu
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          64
On-line CPU(s) list:             0-63
Thread(s) per core:              1
Core(s) per socket:              64
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       ARM

and Python 3.8.5.

I run into an excision halt. My EC2 Ubuntu setup is as follows;

~ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

What is the exact command that was run?

kb ref -i index.idx -g t2g.txt -f1 cdna.fa -f2 intron.fa -c1 cdna_t2c.txt -c2 intron_t2c.txt --workf│
low lamanno -n 4  Mus_musculus.GRCm39.dna.primary_assembly.fa  Mus_musculus.GRCm39.104.gt

Command output (with --verbose flag)

[2021-06-24 17:20:32,944]   ERROR [main] An exception occurred                                              │  19 [                                       0.0%]
Traceback (most recent call last):                                                                          │  20 [                                       0.0%]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/kb_python/main.py", line 856, in main               │  21 [                                       0.0%]
    COMMAND_TO_FUNCTION[args.command](parser, args, temp_dir=temp_dir)                                      │  22 [                                       0.0%]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/kb_python/main.py", line 131, in parse_ref          │  23 [                                       0.0%]
    ref_lamanno(                                                                                            │  24 [                                       0.0%]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/ngs_tools/logging.py", line 62, in inner            │  25 [                                       0.0%]
    return func(*args, **kwargs)                                                                            │  26 [                                       0.0%]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/kb_python/ref.py", line 663, in ref_lamanno         │  27 [                                       0.0%]
    cdna_index_result = kallisto_index(                                                                     │  28 [                                       0.0%]
  File "/home/ubuntu/.local/lib/python3.8/site-packages/kb_python/ref.py", line 212, in kallisto_index      ├───────────────────────────────────────────────────────────────────────────────────────────────────────────
    run_executable(command)                                                                                 │➜  temp tmux ls
  File "/home/ubuntu/.local/lib/python3.8/site-packages/kb_python/dry/__init__.py", line 24, in inner       │0: 1 windows (created Thu Jun 24 17:08:27 2021) (attached)
    return func(*args, **kwargs)                                                                            │➜  temp screen ls
  File "/home/ubuntu/.local/lib/python3.8/site-packages/kb_python/utils.py", line 170, in run_executable    │➜  temp
    p = sp.Popen(                                                                                           │
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__                                            │
    self._execute_child(args, executable, preexec_fn, close_fds,                                            │
  File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child                                     │
    raise child_exception_type(errno_num, err_msg, err_filename)                                            │
OSError: [Errno 8] Exec format error: '/home/ubuntu/.local/lib/python3.8/site-packages/kb_python/bins/linux/│
kallisto/kallisto'

It looks like the shebang issue ? Any help will be appreciate

Lioscro commented 3 years ago

Hi, @akhst7, I have never seen this issue before. Did you install kb through pip or conda?

akhst7 commented 3 years ago

@Lioscro Kb-py and its dependencies were surprisingly successfully compiled and installed though pip3 in the aarch64 running Ubuntu Focal. Doesn't OSError indicate the shebang issue ? I saw this in the stackoverflow https://stackoverflow.com/questions/27606653/oserror-errno-8-exec-format-error. It looks like an easy fix but ....

Lioscro commented 3 years ago

/home/ubuntu/.local/lib/python3.8/site-packages/kb_python/bins/linux/kallisto/kallisto this file is not a shell script, it is a binary, so shebang shouldn't be an issue here. Is it possible to call the binary directly by running

$ /home/ubuntu/.local/lib/python3.8/site-packages/kb_python/bins/linux/kallisto/kallisto
akhst7 commented 3 years ago

I tried that and gave me exec format error. I even went /home/ubuntu/.local/lib/python3.8/site-packages/kb_python/bins/linux/kallisto/ and did ./kallisto but got the same error.

Lioscro commented 3 years ago

Can you try the following?

wget https://github.com/pachterlab/kb_python/raw/master/kb_python/bins/linux/kallisto/kallisto
./kallisto

If that still doesn't work, does the official release downloaded from the kallisto repo (https://github.com/pachterlab/kallisto/releases/download/v0.46.2/kallisto_linux-v0.46.2.tar.gz) work at all?

akhst7 commented 3 years ago

I did try both, 

wget https://github.com/pachterlab/kb_python/raw/master/kb_python/bins/linux/kallisto/kallisto
./kallisto
--2021-07-15 23:25:27--  https://github.com/pachterlab/kb_python/raw/master/kb_python/bins/linux/kallisto/kallisto
Resolving github.com (github.com)... 140.82.113.4
Connecting to github.com (github.com)|140.82.113.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/pachterlab/kb_python/master/kb_python/bins/linux/kallisto/kallisto [following]
--2021-07-15 23:25:28--  https://raw.githubusercontent.com/pachterlab/kb_python/master/kb_python/bins/linux/kallisto/kallisto
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3007008 (2.9M) [application/octet-stream]
Saving to: ‘kallisto’

kallisto               100%[==========================>]   2.87M  --.-KB/s    in 0.08s
curl -o kallisto https://github.com/pachterlab/kallisto/releases/download/v0.46.2/kallisto_linux-v0.46.2.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   635  100   635    0     0   4669      0 --:--:-- --:--:-- --:--:--  4669

and both cases, I gotcommand not found

These binaries are compiled for x86_64 ? If so, it won't run on the aarch machine.

Lioscro commented 3 years ago

Hi, @akhst7, I must have completely missed that part somehow. Yes, that is definitely the issue here. You're going to have to manually compile the kallisto and bustools binaries, and provide the paths to them as the --kallisto and --bustools arguments when running kb.

akhst7 commented 3 years ago

Manual compilation will get ugly at this point. I will just wait till the binary becomes available .