Closed abgutierrez closed 1 year ago
IBM-diksha
commented
1 year ago I have used cos-aspera python library on an intel machine running python 3.6.6. For m1 pro or ppc64le we are looking into it and we will get back to you.
Meanwhile can you please elaborate on your use case here? and also can you please tell me what exactly you have tried till. now ?
abgutierrez
commented
1 year ago Hi @IBM-diksha,
Thanks for the quick response. I am working on a desktop application that is able to interact with Aspera services using the fasp2 protocol. Our team implemented a containerized web version of it months ago. This container has python 3.6.4, ibm-cos-SDK python library (which uses cos-aspera), and the asperatransferd daemon. The previous frontend used is going to be reimplemented in this new desktop app which communicates through rest API with the python backend and is in charge of mounting the files to be uploaded inside the container. In conclusion, all python backend responsible to handle Aspera is working on a container, and the desktop app frontend is intended to manage transfers and mount files to be uploaded.
Let me show you the following diagram, maybe it gives you a better understanding:
abgutierrez
commented
1 year ago I forgot to append build output:
[+] Building 11.1s (21/23)
=> [internal] load build definition from Dockerfile.centos 0.0s
=> => transferring dockerfile: 2.81kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/centos:centos8 3.1s
=> [internal] load build context 0.1s
=> => transferring context: 321.71kB 0.1s
=> [ 1/19] FROM docker.io/library/centos:centos8@sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534bbdee55b 0.0s
=> CACHED [ 2/19] RUN cd /etc/yum.repos.d/ 0.0s
=> CACHED [ 3/19] RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* 0.0s
=> CACHED [ 4/19] RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/ 0.0s
=> CACHED [ 5/19] RUN yum groupinstall -y "Development Tools" && yum install -y wget o 0.0s
=> CACHED [ 6/19] WORKDIR /usr/src 0.0s
=> CACHED [ 7/19] COPY ./aspera.tar.gz ./aspera.tar.gz 0.0s
=> CACHED [ 8/19] RUN tar xvzf aspera.tar.gz && cp -r ./linux-amd64 /opt/aspera && ldconfig /opt/a 0.0s
=> CACHED [ 9/19] WORKDIR /usr/src 0.0s
=> CACHED [10/19] COPY ./frontend ./frontend 0.0s
=> CACHED [11/19] WORKDIR /usr/src/frontend 0.0s
=> CACHED [12/19] RUN npm install && npm run build && npm install -g serve 0.0s
=> CACHED [13/19] COPY redis.conf /usr/local/etc/redis/redis.conf 0.0s
=> CACHED [14/19] WORKDIR /usr/src 0.0s
=> [15/19] COPY ./backend ./backend 0.0s
=> [16/19] WORKDIR /usr/src/backend 0.0s
=> ERROR [17/19] RUN pip3 install --upgrade pip && pip3 install wheel && ls -l && pip3 install 7.8s
------
> [17/19] RUN pip3 install --upgrade pip && pip3 install wheel && ls -l && pip3 install -r ./requirements.txt --no-cache-dir:
#21 0.940 Collecting pip
#21 1.237 Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB)
#21 1.517 Installing collected packages: pip
#21 1.517 Found existing installation: pip 9.0.1
#21 1.554 Uninstalling pip-9.0.1:
#21 2.027 Successfully uninstalled pip-9.0.1
#21 2.506 Successfully installed pip-21.3.1
#21 2.855 Collecting wheel
#21 2.964 Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
#21 2.982 Installing collected packages: wheel
#21 3.001 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#21 3.001 Successfully installed wheel-0.37.1
#21 3.497 Collecting psycopg2-binary
#21 3.580 Downloading psycopg2_binary-2.9.3-cp36-cp36m-manylinux_2_24_aarch64.whl (3.4 MB)
#21 4.156 Collecting ibm-cos-sdk
#21 4.187 Downloading ibm-cos-sdk-2.12.0.tar.gz (55 kB)
#21 4.201 Preparing metadata (setup.py): started
#21 4.333 Preparing metadata (setup.py): finished with status 'done'
#21 4.371 Collecting flask
#21 4.390 Downloading Flask-2.0.3-py3-none-any.whl (95 kB)
#21 4.558 Collecting flask_sslify
#21 4.585 Downloading Flask-SSLify-0.1.5.tar.gz (3.0 kB)
#21 4.589 Preparing metadata (setup.py): started
#21 4.702 Preparing metadata (setup.py): finished with status 'done'
#21 4.743 Collecting flask_cors
#21 4.781 Downloading Flask_Cors-3.0.10-py2.py3-none-any.whl (14 kB)
#21 4.808 Collecting flask_login
#21 4.829 Downloading Flask_Login-0.5.0-py2.py3-none-any.whl (16 kB)
#21 4.859 Collecting flask_migrate
#21 4.884 Downloading Flask_Migrate-3.1.0-py3-none-any.whl (20 kB)
#21 4.912 Collecting flask_sqlalchemy
#21 4.950 Downloading Flask_SQLAlchemy-2.5.1-py2.py3-none-any.whl (17 kB)
#21 5.262 Collecting sqlalchemy
#21 6.784 Downloading SQLAlchemy-1.4.40-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.6 MB)
#21 7.094 Collecting uwsgi
#21 7.117 Downloading uwsgi-2.0.20.tar.gz (804 kB)
#21 7.276 Preparing metadata (setup.py): started
#21 7.404 Preparing metadata (setup.py): finished with status 'done'
#21 7.579 ERROR: Could not find a version that satisfies the requirement cos-aspera (from versions: none)
#21 7.580 ERROR: No matching distribution found for cos-aspera
------
executor failed running [/bin/sh -c pip3 install --upgrade pip && pip3 install wheel && ls -l && pip3 install -r ./requirements.txt --no-cache-dir]: exit code: 1requirements.txt:
psycopg2-binary
ibm-cos-sdk
flask
flask_sslify
flask_cors
flask_login
flask_migrate
flask_sqlalchemy
sqlalchemy
uwsgi
cos-aspera
redis
rq
urllib3
python-dotenv
werkzeug
PyJWT
requests
sendgrid
speedtest-cli
#flask-talisman
@abgutierrez , Support for Aspera in the COS SDKs is currently "legacy": https://github.com/IBM/ibm-cos-sdk-python#legacy-aspera-high-speed-transfer . The aspera team stopped supporting that plugin so it is very unlikely that it would work. So I would suggest you to use Aspera Transfer SDK. you can refer the following: https://developer.ibm.com/apis/catalog/aspera--aspera-transfer-sdk/Introduction/ https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-aspera If I've misunderstood you and you are already using the Aspera Transfer SDKs, you'll need to reach out to Aspera Transfer SDK support.
abgutierrez
commented
1 year ago I understand, thanks for the information.
This part of the project was not developed by me, sorry for so much confusion. I removed cos-aspera from requirements.txt and the build success. Now testing the app I found the following error on the logs related to the package ibm_s3transfer.aspera.manager, this package is being used for transfer management :
File "./app/tasks.py", line 17, in <module>
from ibm_s3transfer.aspera.manager import AsperaConfig, AsperaManagerConfig, \
File "/usr/local/lib/python3.6/site-packages/ibm_s3transfer/aspera/manager.py", line 36, in <module>
from ibm_s3transfer.aspera.futures import AsperaTransferCoordinator
File "/usr/local/lib/python3.6/site-packages/ibm_s3transfer/aspera/futures.py", line 31, in <module>
raise ImportError("Aspera SDK not installed")
ImportError: Aspera SDK not installedIn my case, I want to use ibm-cos-sdk just only to list buckets and objects, so ibm_boto3.client() and ibm_boto3.resource() to get the desired information are enough for me.
Does these methods require cos-aspera dependence? If not, I guess this should work on my code:
IBM-diksha
commented
1 year ago @abgutierrez If your requirement is to list buckets and objects you can use ibm-cos-sdk-python there is no dependency on cos-aspera or ibm_s3transfer.aspera.manager for that. You can refer to the following : https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-python#python-examples-list-objects https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-python#python-examples-list-buckets https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-python
abgutierrez
commented
1 year ago Hi @IBM-diksha ,
Thanks to much for all information, I solved my issue by removing all references to both dependencies (cos-aspera in requirements.txt and ibm_s3transfer.aspera.manager in python implementation for fasp transfer) for now using Aspera Transfer SDK. I'll close this issue.
I'm trying to use cos_aspera on Apple M1 Pro running python 3.6.4. I was reading this issue and found that cos-aspera maven repository doesn't distinguish between m1 and intel, but you can see on pipy repository that mac versions are intel only.
There are no more updates for years and a lack of available architectures, I didn't find cos-aspera source code to compile it or another way to deal with fasp2.
How can we get the source code to compile it on m1,ppc64le, and other available archs? Is there any workaround to interact with aspera fasp2 using Python? Those archs were requested on other issues.