how to import the third packages

langgenius / dify-sandbox

A lightweight, fast, and secure code execution environment that supports multiple programming languages

https://docs.dify.ai/development/backend/sandbox

Apache License 2.0

341 stars 74 forks source link

how to import the third packages #37

Open sharptornadoqsh opened 1 month ago

sharptornadoqsh commented 1 month ago

i follow the faq and build the source source code and tag it to a docker。when i use the docker in dify，i can not import the nump this is the error

it seems there is some permission problems

sharptornadoqsh commented 1 month ago

i follow this setp1 、setp2 to set the config.yaml and zsysnum_linux_amd64.go file https://github.com/langgenius/dify-sandbox/blob/main/FAQ.md#1-why-does-my-python-code-throw-an-exception-like-xxxso-cannot-open-shared-object-file-no-such-file-or-directory

hjlarry commented 1 month ago

try to use numpy 1.x version

sharptornadoqsh commented 1 month ago

您好，您的邮件我已经收到，我会尽快处理

sharptornadoqsh commented 1 month ago

i reaplace the numpy version and set some .so in config.yaml but the problem in picture 2 is still there 2024/08/15 05:25:20 server.go:50: [INFO]initializing python dependencies sandbox... 2024/08/15 05:26:54 env.go:30: [WARN]python lib path /usr/lib/python3.10 is not available 2024/08/15 05:26:54 env.go:30: [WARN]python lib path /usr/lib/python3 is not available 2024/08/15 05:26:54 env.go:30: [WARN]python lib path /run/systemd/resolve/stub-resolv.conf is not available 2024/08/15 05:26:54 env.go:30: [WARN]python lib path /run/resolvconf/resolv.conf is not available 2024/08/15 05:26:54 server.go:55: [INFO]python dependencies sandbox initialized [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middlew

hjlarry commented 1 month ago

it seems your python is not in the /usr/lib/python3.10 or /usr/lib/python3, try to fix that first.

sharptornadoqsh commented 1 month ago

it seems your python is not in the /usr/lib/python3.10 or /usr/lib/python3, try to fix that first.

you are right i modify the config.yaml

but new problems

this is the error in dify web： Node 代码执行 2 run failed: READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following: The Python version is: Python3.10 from "/usr/local/bin/python3.10" The NumPy version is: "1.23.5" and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help. Original error was: libpthread.so.0: cannot open shared object file: No such file or directory ne 48, in File "", line 4, in File "/usr/local/lib/python3.10/site-packages/numpy/init.py", line 140, in from . import core File "/usr/local/lib/python3.10/site-packages/numpy/core/init.py", line 49, in raise ImportError(msg) ImportError: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following: The Python version is: Python3.10 from "/usr/local/bin/python3.10" The NumPy version is: "1.23.5" and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help. Original error was: libpthread.so.0: cannot open shared object file: No such file or directory error: exit status 255

this is the sandbox docker log ![Uploading 1.png…]()

sharptornadoqsh commented 1 month ago

2024/08/15 06:23:49 setup.go:29: [INFO]initializing nodejs runner environment... 2024/08/15 06:23:50 setup.go:85: [INFO]nodejs runner environment initialized 2024/08/15 06:23:50 setup.go:31: [INFO]initializing python runner environment... 2024/08/15 06:23:50 setup.go:43: [INFO]python runner environment initialized 2024/08/15 06:23:50 config.go:96: [INFO]network has been enabled 2024/08/15 06:23:50 config.go:112: [INFO]using https proxy: http://ssrf_proxy:3128 2024/08/15 06:23:50 config.go:121: [INFO]using http proxy: http://ssrf_proxy:3128 2024/08/15 06:23:50 server.go:20: [INFO]config init success 2024/08/15 06:23:50 server.go:26: [INFO]runner dependencies init success 2024/08/15 06:23:50 server.go:42: [INFO]installing python dependencies... 2024/08/15 06:23:52 setup.go:114: [INFO]Requirement already satisfied: numpy==1.23.5 in /usr/local/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (1.23.5) 2024/08/15 06:23:52 setup.go:114: [INFO]Requirement already satisfied: pypdf==3.17.4 in /usr/local/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (3.17.4) 2024/08/15 06:23:52 setup.go:114: [INFO]Requirement already satisfied: markdown==3.5.2 in /usr/local/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (3.5.2) 2024/08/15 06:23:53 setup.go:135: [INFO]Python dependency installed: numpy 1.23.5 2024/08/15 06:23:53 setup.go:135: [INFO]Python dependency installed: pypdf 3.17.4 2024/08/15 06:23:53 setup.go:135: [INFO]Python dependency installed: markdown 3.5.2 2024/08/15 06:23:53 server.go:48: [INFO]python dependencies installed 2024/08/15 06:23:53 server.go:50: [INFO]initializing python dependencies sandbox... 2024/08/15 06:25:13 server.go:55: [INFO]python dependencies sandbox initialized 2024/08/15 06:25:13 cocrrent.go:31: [INFO]setting max requests to 50 2024/08/15 06:25:13 cocrrent.go:13: [INFO]setting max workers to 4

hjlarry commented 1 month ago

follow the FAQ1 add the libpthread.so.0

sharptornadoqsh commented 1 month ago

follow the FAQ1 add the libpthread.so.0

thank you。i find the problem THE FAQ documnet 's system call in numpy is wrong

hjlarry commented 1 month ago

the document is to show how to resolve the problem, but the final result is based on different systems

Korayem commented 1 week ago

I want to use sandbox to create charts using pandas and matplotlib but returns errors because packages aren't installed line 1, in <module>\nModuleNotFoundError: No module named 'matplotlib'

So here's what I tried to do in dify local git clone (I am running latest dify 0.7.3)

Create a custom Dockerfile: Create a new Dockerfile in your /dif/docker directory. Named it sandbox.dockerfile.

# sandbox.dockerfile
FROM langgenius/dify-sandbox:latest

# Copy the requirements.txt file into the container
COPY sandbox-requirements.txt /dependencies/python-requirements.txt

# Install the dependencies
RUN pip3 install --no-cache-dir -r /dependencies/python-requirements.txt

create docker/sandbox-requirements.txt
```
matplotlib
pandas
```

Modify the docker-compose.yaml to use the custom Dockerfile: Update the sandbox service to build from the custom Dockerfile.

// ... existing code ...
sandbox:
 build:
   context: .
   dockerfile: sandbox.dockerfile
 restart: always
 environment:
   # The DifySandbox configurations
   # Make sure you are changing this key for your deployment with a strong key.
   # You can generate a strong key using `openssl rand -base64 42`.
   API_KEY: ${SANDBOX_API_KEY:-dify-sandbox}
   GIN_MODE: ${SANDBOX_GIN_MODE:-release}
   WORKER_TIMEOUT: ${SANDBOX_WORKER_TIMEOUT:-15}
   ENABLE_NETWORK: ${SANDBOX_ENABLE_NETWORK:-true}
   HTTP_PROXY: ${SANDBOX_HTTP_PROXY:-http://ssrf_proxy:3128}
   HTTPS_PROXY: ${SANDBOX_HTTPS_PROXY:-http://ssrf_proxy:3128}
   SANDBOX_PORT: ${SANDBOX_PORT:-8194}
 volumes:
   - ./volumes/sandbox/dependencies:/dependencies
   - ./requirements.txt:/app/requirements.txt  # Mount the requirements.txt file
 networks:
   - ssrf_proxy_network
// ... existing code ...

Build and run the updated service: Run the following commands to build and start the updated sandbox service.
```
docker-compose build sandbox
docker-compose up sandbox
```

This indeed fixes pandas and matplotlib error but for some reason numpy (a dependency of both pandas and matplotlib) is still not found and throws an error

Traceback (most recent call last):
File \"/var/sandbox/sandbox-python/tmp/a8bb7065_f87e_4c56_a91e_3d3d6aa23509.py\", line 48, in <module>
File \"<string>\", line 1, in <module>
File \"/usr/local/lib/python3.10/site-packages/pandas/__init__.py\", line 19, in <module>
raise ImportError(ImportError:
 Unable to import required dependencies: numpy
  Error importing numpy: you should not try to import numpy from its source directory; please exit the numpy source tree, and relaunch your python interpreter from there.

Notice that it read that pandas package is installed locally this time, but it can't find its dependency numpy, though from the logs when building sandbox with custom python-requirements.txt, numpy did get installed

To further verify, I checked sandox container files python packages in the path /usr/local/lib/python3.10/site-packages

I also ran

docker exec -it 184442099b89a187fd8fabf983d81e202ecc04ab204883a495792ba67418badb /bin/sh
pip3 list | grep numpy

And verified it's installed numpy 2.1.0

I did add numpy to python-requirements.txt but still the same error showed up

I feel I am so close but don't know what else to do

Thanks in advance

Korayem commented 1 week ago

After further trials, I added the following to config.yaml

python_lib_path:
  - /usr/local/lib/python3.10
  - /usr/local/lib/python3.10/site-packages

Now I get this error 🙃

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

 https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

 * The Python version is: Python3.10 from \"/usr/local/bin/python3\"
 * The NumPy version is: \"2.1.0\"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libgcc_s.so.1: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File \"/var/sandbox/sandbox-python/tmp/d5600e89_9809_498c_89ac_372be73bcb7e.py\", line 48, in <module>
 File \"<string>\", line 1, in <module>
 File \"/usr/local/lib/python3.10/site-packages/matplortError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

 https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

 * The Python version is: Python3.10 from \"/usr/local/bin/python3\"
 * The NumPy version is: \"2.1.0\"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libgcc_s.so.1: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File \"/var/sandbox/sandbox-python/tmp/d5600e89_9809_498c_89ac_372be73bcb7e.py\", line 48, in <module>
 File \"<string>\", line 1, in <module>
 File \"/usr/local/lib/python3.10/site-packages/matplotlib/__init__.py\", line 159, in <module>
 from . import _api, _version, cbook, _docstring, rcsetup
 File \"/usr/local/lib/python3.10/site-packages/matplotlib/cbook.py\", line 24, in <module>
 import numpy as np
 File \"/usr/local/lib/python3.10/site-packages/numpy/__init__.py\", line 119, in <module>
 raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
 its source directory; please exit the numpy source tree, and relaunch
 your python interpreter from there.
error: exit status 255

Korayem commented 1 week ago

Realizing that libgcc_s.so.1: cannot open shared object file: No such file or directory requires installing apt-get to linux for numpy, I went on another route of using plotly which doesnt have complicated dependencies. It worked great, but can't write generated image to chart due to file restrictions. Any idea how can I prompt code interpreter to generate an image of the chart and return it to Dify Agent?

yuweimian-shy commented 1 week ago

After further trials, I added the following to config.yaml

python_lib_path:
  - /usr/local/lib/python3.10
  - /usr/local/lib/python3.10/site-packages

Now I get this error 🙃

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

 https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

 * The Python version is: Python3.10 from \"/usr/local/bin/python3\"
 * The NumPy version is: \"2.1.0\"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libgcc_s.so.1: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File \"/var/sandbox/sandbox-python/tmp/d5600e89_9809_498c_89ac_372be73bcb7e.py\", line 48, in <module>
 File \"<string>\", line 1, in <module>
 File \"/usr/local/lib/python3.10/site-packages/matplortError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

 https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

 * The Python version is: Python3.10 from \"/usr/local/bin/python3\"
 * The NumPy version is: \"2.1.0\"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: libgcc_s.so.1: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File \"/var/sandbox/sandbox-python/tmp/d5600e89_9809_498c_89ac_372be73bcb7e.py\", line 48, in <module>
 File \"<string>\", line 1, in <module>
 File \"/usr/local/lib/python3.10/site-packages/matplotlib/__init__.py\", line 159, in <module>
 from . import _api, _version, cbook, _docstring, rcsetup
 File \"/usr/local/lib/python3.10/site-packages/matplotlib/cbook.py\", line 24, in <module>
 import numpy as np
 File \"/usr/local/lib/python3.10/site-packages/numpy/__init__.py\", line 119, in <module>
 raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
 its source directory; please exit the numpy source tree, and relaunch
 your python interpreter from there.
error: exit status 255

I also have this problem when I use numpy and other libraries that depend on numpy, do you have any solutions for it now?

Korayem commented 1 week ago

I also have this problem when I use numpy and other libraries that depend on numpy, do you have any solutions for it now?

@yuweimian-shy Check my previous reply https://github.com/langgenius/dify-sandbox/issues/37#issuecomment-2326520238

AFAIK from the FAQ, running OS level packages using apt-get requires further configuration for syscall that I wasn't free to try out. So I reverted to plotly but I can't generate a chart image and send it back to Dify to render. Requires investigation with plotly and how to send generated chart image back.

nhha1602 commented 1 week ago

I tried to get system call number .... I run main and below is log of main:

2024/09/07 16:45:40 setup.go:29: [INFO]initializing nodejs runner environment...
2024/09/07 16:45:40 setup.go:85: [INFO]nodejs runner environment initialized
2024/09/07 16:45:40 setup.go:33: [INFO]initializing python runner environment...
2024/09/07 16:45:40 setup.go:50: [INFO]python runner environment initialized
2024/09/07 16:45:40 config.go:96: [INFO]network has been enabled
2024/09/07 16:45:40 server.go:20: [INFO]config init success
2024/09/07 16:45:40 server.go:26: [INFO]runner dependencies init success
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /health                   --> github.com/langgenius/dify-sandbox/internal/controller.Setup.func1 (5 handlers)
2024/09/07 16:45:40 cocrrent.go:31: [INFO]setting max requests to 50
2024/09/07 16:45:40 server.go:47: [INFO]installing python dependencies...
2024/09/07 16:45:40 cocrrent.go:13: [INFO]setting max workers to 4
2024/09/07 16:45:40 server.go:53: [INFO]python dependencies installed
2024/09/07 16:45:40 server.go:55: [INFO]initializing python dependencies sandbox...
[GIN-debug] POST   /v1/sandbox/run           --> github.com/langgenius/dify-sandbox/internal/controller.RunSandboxController (8 handlers)
[GIN-debug] GET    /v1/sandbox/dependencies  --> github.com/langgenius/dify-sandbox/internal/controller.GetDependencies (6 handlers)
[GIN-debug] POST   /v1/sandbox/dependencies/update --> github.com/langgenius/dify-sandbox/internal/controller.UpdateDependencies (6 handlers)
[GIN-debug] GET    /v1/sandbox/dependencies/refresh --> github.com/langgenius/dify-sandbox/internal/controller.RefreshDependencies (6 handlers)
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
[GIN-debug] Listening and serving HTTP on :8194
2024/09/07 16:46:27 env.go:30: [WARN]python lib path /etc/ssl/certs/ca-certificates.crt is not available
2024/09/07 16:46:27 server.go:60: [INFO]python dependencies sandbox initialized

And I run below command but there is no printed syscalls:

go run cmd/test/syscall_dig/main.go
Following syscalls are required:

Please help.