I cannot load "from python" import numpy and error: cannot typecheck the program for empty variables

[Jordi-Valls]

Dear all, First I want to congratulate the codon community to develop tools that make python more efficient.

Now I want to explain a bit about my limitations in executing codon with my script. I read detailed the documentation of https://docs.exaloop.io/codon/interoperability/python, but unfortunately, I cannot execute my python script.

I use different python libraries. This is the first part of my code:

import multiprocessing import pandas as pd import numpy as np from numpy.linalg import norm

def calculate_new_data1(drug): A = np.load(''_p_visit_array.npy') return A

And as you can see, I get the same errors as other users:

load_libraries.py:1:8-23: error: no module named 'multiprocessing' load_libraries.py:2:8-14: error: no module named 'pandas' load_libraries.py:3:8-13: error: no module named 'numpy' load_libraries.py:4:6-18: error: no module named 'numpy.linalg' load_libraries.py:7:9-167: error: no module named 'np' load_libraries.py:8:12-13: error: name 'A' is not defined

I would appreciate detailed steps in order to execute this script in condon, because the other part of the script is a loop that uses these libraries, so if I can load the libraries, I think that I could execute my script.

I will enumerate the questions:

1- I have different python versions on my laptop, and I would like to import one of them. When I do "from python" which python takes? How can I change "from python" to another python on my laptop? I have to change my PYTHONPATH? If this is the key, please, could you explain to me how to do it? The path must to go in the folder where are all the libraries installed?

2- My second question is about "export CODON_PYTHON=/path/to/libpython.X.Y.so", How can I have to apply this? Do I have to put this in my ./bash_profile? I have to attach different libraries such as pandas, numpy, how can I import the correct python version with export CODON_PYTHON=, In my opinion, a detailed example could be useful.

3- I don't know how works the @python, I understand that can be used to write the python code to codon directly, but I don't know how it works.

Definitely, as you can see maybe my questions are a bit basic, but If some of you could answer my questions with a bit of detail (step by step) maybe more bioinformatics non-familiar with codon could start.

By the way, I was able to execute the basic code provided in the documentation:

from time import time

def fib(n): return n if n < 2 else fib(n - 1) + fib(n - 2)

t0 = time() ans = fib(40) t1 = time() print(f'Computed fib(40) = {ans} in {t1 - t0} seconds.')

But this example is difficult to apply when you need to import libraries...

Thanks for your help and time.

Best,

Jordi

[Jordi-Valls]

Finally, I could execute this first part of the script, if I export this "export CODON_PYTHON=path/to/libpython3.8.so". The following script allows you to find where is located this file on your laptop: https://github.com/exaloop/codon/blob/develop/test/python/find-python-library.py

Then, I could load all libraries.

Now I've a problem with variables. Why if I declare an empty variable ( dict_all_sim = {} ) I get the following error?

error: cannot typecheck the program

Here I show you the script that report the following error

from python import multiprocessing from python import pandas as pd from python import numpy as np from python import numpy.linalg as norm

def calculate_new_data1(drug): A = np.load('/home/jordi/multiscale_interactome/multiscale-interactome-master/HPC_CEP/performance_output/' + drug.strip().replace('-', '') + '_p_visit_array.npy') return A

dict_all_sim = {}

[elisbyberi]

@Jordi-Valls If I were to answer your questions, I would reproduce the same documentation found at: https://docs.exaloop.io/codon/interoperability/python The previous sentence doesn't sound helpful, but it would be more effective if you provided real examples and identified the specific errors or areas where you find it difficult to follow the documentation. However, it seems like you have already answered them yourself, haven't you?

For example, Codon is strongly typed. The type of dict_all_sim = {} is not predefined, nor can it be deduced.

[Jordi-Valls]

Hello @elisbyberi, the error that I had with the dict_all_sim = {} was [error: cannot typecheck the program for empty variables]. It is true that I've to type the variable, however, I don't know how to do it. The dictionary has a key as a string, and then I have a dataframe which is included in the key. I tried to type this variable like dict_all_sim: Dict[str,pd.DataFrame(str, float)], but it doesn't work, I have the same error. So how I can type a dictionary including a dataframe inside?

On the other hand, I found that Numpy is not already well applied in codon right? because I tried to execute the functions "dot" and import numpy.linalg as norm from numpy, And I get the error "No Module is callable". So I think that I've to wait for future updates right?

Finally, thanks for helping me, I appreciate it.

[arshajii]

Hi @Jordi-Valls -- if you have a data frame from Python, its type in Codon will be pyobj. Can you try Dict[str, pyobj] and see if that works?

[inumanag]

Hi @Jordi-Valls

import multiprocessing import pandas as pd import numpy as np from numpy.linalg import norm

You will need to use from python import XYZ: we still do not support pandas and numpy natively. multiprocessing most likely won't work at all.

1- I have different python versions on my laptop, and I would like to import one of them. When I do "from python" which python takes? How can I change "from python" to another python on my laptop? I have to change my PYTHONPATH? If this is the key, please, could you explain to me how to do it? The path must to go in the folder where are all the libraries installed?

Different Python versions come with different libpython.so libraries: you need to point to the one you want. Also, you might need to set up PYTHONHOME as well.

2- My second question is about "export CODON_PYTHON=/path/to/libpython.X.Y.so", How can I have to apply this? Do I have to put this in my ./bash_profile? I have to attach different libraries such as pandas, numpy, how can I import the correct python version with export CODON_PYTHON=, In my opinion, a detailed example could be useful.

The script you used in the answer is the correct way to go. Alternatively, locate libpython.so or find / grep libpython.so (or similar) might help.

Hello @elisbyberi, the error that I had with the dict_all_sim = {} was [error: cannot typecheck the program for empty variables]. It is true that I've to type the variable, however, I don't know how to do it. The dictionary has a key as a string, and then I have a dataframe which is included in the key.

If you do not use that dict anywhere, Codon cannot deduce its type. Maybe try dict_all_sim: Dict[str, pyobj] = {} instead.

[Jordi-Valls]

Thanks to everybody for the answers. Finally, I could solve it! However, as I need numpy for my script, and pandas, and it seems that codon has not already optimized these python libraries, my script is still faster if I compare it with the version in codon. I think that codon is promising and I will happily apply the new incoming updates!