Test deployment of the API and GUI on the new VM #9

Closed kinow closed 2 months ago

In GitLab by @mcastril on Apr 1, 2022, 18:33

Now that the VM is available, @jberlin could you proceed with testing the implementation of the API and the GUI?

[x] Autosubmit GUI - NoAPI
[x] Autosubmit GUI - Distributed - (Consuming API from web server bscweb4)
[x] Autosubmit GUI - Standalone - (Consuming API from same VM )

If you see something that it's not working or not documented please report it here.

@wuruchi

In GitLab by @jberlin on Apr 4, 2022, 08:45

Hi I tried to log in but I get permission denied:

bscearth357 /home/Earth/jberlin  > ssh jberlin@84.88.189.213
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

I guess either my key is wrong or effectively there are some issues with the permissions

In GitLab by @mcastril on Apr 4, 2022, 11:13

Hi @jberlin, did you contact @kserrade ?

I had an issue with the permission:

mcastril@bscearth396:~> ssh -i .ssh/id_rsa.pub rocky@84.88.189.213
Warning: Permanently added '84.88.189.213' (ECDSA) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '.ssh/id_rsa.pub' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key ".ssh/id_rsa.pub": bad permissions
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

But after doing chmod 640 id_rsa.pub it worked. Maybe it's not the same because you didn't have the same error message.

In GitLab by @jberlin on Apr 4, 2022, 11:28

Hi @mcastril , I just managed to do it:

bscearth357 /home/Earth/jberlin  > ssh -i .ssh/id_rsa.pub rocky@84.88.189.213
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Mon Apr  4 09:16:15 2022 from 84.88.185.148
[rocky@vm ~]$ ll
total 0
[rocky@vm ~]$ pwd
/home/rocky

the way I was trying was wrong (I was trying directly with my user).

Saludos!

In GitLab by @jberlin on Apr 4, 2022, 12:56

I got some permissions issues and also because the machine lack of the base software such as git, python I had to install them manually, maybe @wuruchi concerning the Python version, should be indicated in the installation which version to install, I also got some permissions issues:

[root@vm autosubmitreact]# npm install
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!

> canvas@2.7.0 install /home/rocky/development/autosubmitreact/node_modules/canvas
> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using request for node-pre-gyp https download 
node-pre-gyp WARN Pre-built binaries not installable for canvas@2.7.0 and node@14.18.2 (node-v83 ABI, glibc) (falling back to source compile with node-gyp) 
node-pre-gyp WARN Hit error EACCES: permission denied, mkdir '/home/rocky/development/autosubmitreact/node_modules/canvas/build' 
gyp WARN EACCES current user ("rocky") does not have permission to access the dev dir "/root/.cache/node-gyp/14.18.2"
gyp WARN EACCES attempting to reinstall using temporary dev dir "/home/rocky/development/autosubmitreact/node_modules/canvas/.node-gyp"
gyp WARN install got an error, rolling back install
gyp WARN install got an error, rolling back install
gyp ERR! configure error 
gyp ERR! stack Error: EACCES: permission denied, mkdir '/home/rocky/development/autosubmitreact/node_modules/canvas/.node-gyp'
gyp ERR! System Linux 4.18.0-348.20.1.el8_5.x86_64
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/home/rocky/development/autosubmitreact/node_modules/canvas/build/Release/canvas.node" "--module_name=canvas" "--module_path=/home/rocky/development/autosubmitreact/node_modules/canvas/build/Release" "--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v83" "--python=/usr/bin/python3"
gyp ERR! cwd /home/rocky/development/autosubmitreact/node_modules/canvas
gyp ERR! node -v v14.18.2
gyp ERR! node-gyp -v v5.1.0
gyp ERR! not ok 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/home/rocky/development/autosubmitreact/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/home/rocky/development/autosubmitreact/node_modules/canvas/build/Release --napi_version=8 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83 --python=/usr/bin/python3' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/home/rocky/development/autosubmitreact/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:400:28)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1058:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:293:5)
node-pre-gyp ERR! System Linux 4.18.0-348.20.1.el8_5.x86_64
node-pre-gyp ERR! command "/usr/bin/node" "/home/rocky/development/autosubmitreact/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /home/rocky/development/autosubmitreact/node_modules/canvas
node-pre-gyp ERR! node -v v14.18.2
node-pre-gyp ERR! node-pre-gyp -v v0.15.0
node-pre-gyp ERR! not ok 
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js configure --fallback-to-build --module=/home/rocky/development/autosubmitreact/node_modules/canvas/build/Release/canvas.node --module_name=canvas --module_path=/home/rocky/development/autosubmitreact/node_modules/canvas/build/Release --napi_version=8 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v83 --python=/usr/bin/python3' (1)
npm WARN @babel/plugin-bugfix-v8-spread-parameters-in-optional-chaining@7.13.12 requires a peer of @babel/core@^7.13.0 but none is installed. You must install peer dependencies yourself.
npm WARN react-dom@16.14.0 requires a peer of react@^16.14.0 but none is installed. You must install peer dependencies yourself.
npm WARN react-web-notification@0.8.0 requires a peer of react@^16.8.6 but none is installed. You must install peer dependencies yourself.
npm WARN tsutils@3.21.0 requires a peer of typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta but none is installed. You must install peer dependencies yourself.
npm WARN vis-data@6.6.1 requires a peer of vis-util@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/watchpack-chokidar2/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/webpack-dev-server/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! canvas@2.7.0 install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the canvas@2.7.0 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-04-04T10_45_00_227Z-debug.log

I think these are because is a git directory but not sure

In GitLab by @wuruchi on Apr 4, 2022, 14:17

Hello @jberlin

maybe @wuruchi concerning the Python version, should be indicated in the installation which version to install

Sure, I'll add it.

Regarding npm, I don't have it installed in the production server bscesweb04 because I build the deployment package (npm run build) in my local workstation or any other machine where I con freely work with npm.

In GitLab by @jberlin on Apr 4, 2022, 15:01

I did some research , and adding the following flag --unsafe-perm seems to fix the install issue with the permissions:

[root@vm autosubmitreact]# npm install --unsafe-perm npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!

[root@vm autosubmitreact]# npm install --unsafe-perm
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!

> canvas@2.7.0 install /home/rocky/development/autosubmitreact/node_modules/canvas
> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using request for node-pre-gyp https download 
[canvas] Success: "/home/rocky/development/autosubmitreact/node_modules/canvas/build/Release/canvas.node" is installed via remote
npm WARN @babel/plugin-bugfix-v8-spread-parameters-in-optional-chaining@7.13.12 requires a peer of @babel/core@^7.13.0 but none is installed. You must install peer dependencies yourself.
npm WARN react-dom@16.14.0 requires a peer of react@^16.14.0 but none is installed. You must install peer dependencies yourself.
npm WARN react-web-notification@0.8.0 requires a peer of react@^16.8.6 but none is installed. You must install peer dependencies yourself.
npm WARN tsutils@3.21.0 requires a peer of typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta but none is installed. You must install peer dependencies yourself.
npm WARN vis-data@6.6.1 requires a peer of vis-util@^4.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@2.3.2 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@2.3.2: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/watchpack-chokidar2/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.13 (node_modules/webpack-dev-server/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.13: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 52 packages from 33 contributors and audited 2090 packages in 20.371s

141 packages are looking for funding
  run `npm fund` for details

found 80 vulnerabilities (3 low, 46 moderate, 27 high, 4 critical)
  run `npm audit fix` to fix them, or `npm audit` for details

In GitLab by @jberlin on Apr 4, 2022, 15:09

I managed to set app up, but connecting from outside gives timeout:

You can now view autosubmitgui in the browser.

  Local:            http://localhost:3000/autosubmitapp
  On Your Network:  http://192.168.0.56:3000/autosubmitapp

Note that the development build is not optimized.
To create a production build, use npm run build.

trying to sorting out what is the problem

In GitLab by @wuruchi on Apr 4, 2022, 15:56

Hello @mcastril

To properly test the deployment of Autosubmit GUI, @jberlin needs that there is a web server (Apache App server, Nginx, or similar) already running and configured in the VM. Usually, this is a job for the admin, since it involves changing server settings that could involve some security issues.

In GitLab by @mcastril on Apr 4, 2022, 16:22

Yes @wuruchi, if Apache is not installed, I think it's better to ask it to sysadmin.

@kserrade I think it's you who should ask for it.

In GitLab by @mcastril on Apr 4, 2022, 16:23

Yes it's true that maybe you don't need npm in the remote where you can deploy your locally build packages.

Anyway, happy to know that you could already install the API.

In GitLab by @kserrade on Apr 5, 2022, 09:35

Hello, sysadmin does not administer this VM. We need to do it ourselves.

Should I install the webserver?

In GitLab by @jberlin on Apr 5, 2022, 09:39

Hi @kserrade , I think yes, we should have the same scenario as current webserver where the app is hosted. so we can use this one as development environment

In GitLab by @kserrade on Apr 5, 2022, 09:53

I've tried to install apache but when launching, port 80 was already in use.

I've removed the webserver and it seems to work (connection comes from nodejs)

[root@vm rocky]# fuser 80/tcp
80/tcp:              20415
[root@vm rocky]# ls -l /proc/20415/exe
lrwxrwxrwx. 1 root root 0 Apr  4 15:13 /proc/20415/exe -> /usr/bin/node

http://84.88.189.213/autosubmitapp/

In GitLab by @jberlin on Apr 5, 2022, 10:01

Hi @kserrade , great ! before from outside the machine it was giving timeout when doing npm start inside the autosubmitapp folder

In GitLab by @kserrade on Apr 5, 2022, 10:03

I've also had to open port 80 in the OpenStack interface.

In GitLab by @jberlin on Apr 5, 2022, 10:20

I think we should include this in the README or wiki since is a fast way to troubleshoot this type of problems on deploying the app.

In GitLab by @jberlin on Apr 5, 2022, 13:28

Hi @kserrade , now for some reason I am unable to connect to the machine, yesterday from the office was working ok:

bscearth357 /home/Earth/jberlin  > ssh -i .ssh/id_rsa.pub rocky@84.88.189.213 -X
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

maybe the latest changes done had an impact on the ssh keys ?

In GitLab by @kserrade on Apr 5, 2022, 14:43

It's really strange. Connection is different whether I use bscearth408 or bscearth357.

Let me check.

In GitLab by @kserrade on Apr 5, 2022, 14:44

Can you connect only using

ssh -X rocky@84.88.189.213

In GitLab by @jberlin on Apr 5, 2022, 14:48

no, same error:

bscearth357 /home/Earth/jberlin  > ssh -X rocky@84.88.189.213
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

In GitLab by @jberlin on Apr 6, 2022, 12:15

Hi @wuruchi , in order to have access to bscesweb04, should I create a petition to helpdesk right ? thanks

In GitLab by @wuruchi on Apr 6, 2022, 14:09

Hello @jberlin

Yes, although I don't remember exactly how we did it the first time.

I first connect to bscesweb04 as my user, and then I connect again (still in bscesweb04) as webadmin.

In GitLab by @mcastril on Apr 6, 2022, 17:08

Can you access now @jberlin ?

In GitLab by @mcastril on Apr 6, 2022, 17:08

I don't think you can do it yourself, I think @kserrade asked for @wuruchi access right?

In GitLab by @kserrade on Apr 6, 2022, 17:09

Yes, I think I need to ask for it.

In GitLab by @jberlin on Apr 6, 2022, 17:11

I tried but I could not

bscearth357 /home/Earth/jberlin  > ssh jberlin@bscesweb04
Password: 
Permission denied (publickey,keyboard-interactive).

In GitLab by @mcastril on Apr 7, 2022, 10:16

@kserrade could you inform us when the webserver is installed? Thanks!

In GitLab by @kserrade on Apr 7, 2022, 10:27

Sorry, I did not understand. Apache is installed but couldn't be started due to the usage of port 80 by node.

As soon as you stop the application, you can start the webserver with

[root@vm rocky]# systemctl start httpd

In GitLab by @jberlin on Apr 7, 2022, 12:51

Hi, I just killed the instance of nodejs, and launched the apache server I think is wise to use apache to host the node js app , we can also use it for the API

In GitLab by @jberlin on Apr 7, 2022, 15:38

I managed to deploy the app under Apache server, now I will continue with the Autosubmit API.

In GitLab by @jberlin on Apr 8, 2022, 10:09

marked the checklist item Autosubmit GUI - NoAPI as completed

In GitLab by @jberlin on Apr 8, 2022, 12:20

if we are using a url different from the production one, we are unable to login, I guess this be default restrict us to only set AUTHENTICATION=false in development environments like the one we will mount in the VM.

In GitLab by @wuruchi on Apr 8, 2022, 13:16

Hello @jberlin

Support should authorize your URL to ask for CAS credentials.

In GitLab by @mcastril on Apr 8, 2022, 14:03

Thanks @jberlin . Can you ask for it? I think you need to CC @kserrade

In GitLab by @kserrade on Apr 8, 2022, 14:23

I'll ask Operations.

In GitLab by @jberlin on Apr 8, 2022, 15:44

also, I think we should check if this current mechanism of authentication will work with other HPC infraestructures for the upcoming future.

In GitLab by @mcastril on Apr 8, 2022, 15:52

I think it's more important to ensure that we can use the GUI without CAS too, at this point.

Once that @wuruchi makes the necessary changes, it would be good to test it with and without CAS.

In GitLab by @jberlin on Apr 8, 2022, 17:51

Hi I tried to install the autosubmit API but I am getting some errors on the pip installer:

source  /var/www/html/autosubmitapi/venv27/bin/activate
(venv27) [root@vm autosubmitapi]# pip2 install autosubmit_api

I got this error:

Collecting autosubmit_api
  Using cached autosubmit_api-1.0.17-py2-none-any.whl (357 kB)
Requirement already satisfied: graphviz==0.13 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (0.13)
Requirement already satisfied: portalocker==0.5.7 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (0.5.7)
Collecting regex
  Using cached regex-2022.1.18.tar.gz (382 kB)
Requirement already satisfied: gunicorn==19.9.0 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (19.9.0)
Requirement already satisfied: Flask-RESTful==0.3.7 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (0.3.7)
Requirement already satisfied: Flask==1.1.1 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (1.1.1)
Requirement already satisfied: Flask-Cors==3.0.8 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (3.0.8)
Requirement already satisfied: bscearth.utils==0.5.2 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (0.5.2)
Requirement already satisfied: requests==2.22.0 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (2.22.0)
Requirement already satisfied: numpy in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (1.16.6)
Requirement already satisfied: paramiko==1.15.0 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (1.15.0)
Requirement already satisfied: networkx==2.2 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (2.2)
Requirement already satisfied: pydotplus==2.0.2 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (2.0.2)
Requirement already satisfied: Flask-Jsonpify==1.5.0 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (1.5.0)
Requirement already satisfied: mock==3.0.5 in ./venv27/lib/python2.7/site-packages (from autosubmit_api) (3.0.5)
Requirement already satisfied: six>=1.3.0 in ./venv27/lib/python2.7/site-packages (from Flask-RESTful==0.3.7->autosubmit_api) (1.16.0)
Requirement already satisfied: pytz in ./venv27/lib/python2.7/site-packages (from Flask-RESTful==0.3.7->autosubmit_api) (2022.1)
Requirement already satisfied: aniso8601>=0.82 in ./venv27/lib/python2.7/site-packages (from Flask-RESTful==0.3.7->autosubmit_api) (9.0.1)
Requirement already satisfied: Werkzeug>=0.15 in ./venv27/lib/python2.7/site-packages (from Flask==1.1.1->autosubmit_api) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in ./venv27/lib/python2.7/site-packages (from Flask==1.1.1->autosubmit_api) (1.1.0)
Requirement already satisfied: click>=5.1 in ./venv27/lib/python2.7/site-packages (from Flask==1.1.1->autosubmit_api) (7.1.2)
Requirement already satisfied: Jinja2>=2.10.1 in ./venv27/lib/python2.7/site-packages (from Flask==1.1.1->autosubmit_api) (2.11.3)
Requirement already satisfied: pyparsing in ./venv27/lib/python2.7/site-packages (from bscearth.utils==0.5.2->autosubmit_api) (2.4.7)
Requirement already satisfied: python-dateutil in ./venv27/lib/python2.7/site-packages (from bscearth.utils==0.5.2->autosubmit_api) (2.8.2)
Requirement already satisfied: idna<2.9,>=2.5 in ./venv27/lib/python2.7/site-packages (from requests==2.22.0->autosubmit_api) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./venv27/lib/python2.7/site-packages (from requests==2.22.0->autosubmit_api) (1.25.11)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./venv27/lib/python2.7/site-packages (from requests==2.22.0->autosubmit_api) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./venv27/lib/python2.7/site-packages (from requests==2.22.0->autosubmit_api) (2021.10.8)
Requirement already satisfied: pycrypto!=2.4,>=2.1 in ./venv27/lib/python2.7/site-packages (from paramiko==1.15.0->autosubmit_api) (2.6.1)
Requirement already satisfied: ecdsa>=0.11 in ./venv27/lib/python2.7/site-packages (from paramiko==1.15.0->autosubmit_api) (0.17.0)
Requirement already satisfied: decorator>=4.3.0 in ./venv27/lib/python2.7/site-packages (from networkx==2.2->autosubmit_api) (4.4.2)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in ./venv27/lib/python2.7/site-packages (from mock==3.0.5->autosubmit_api) (1.0.2)
Requirement already satisfied: MarkupSafe>=0.23 in ./venv27/lib/python2.7/site-packages (from Jinja2>=2.10.1->Flask==1.1.1->autosubmit_api) (1.1.1)
Building wheels for collected packages: regex
  Building wheel for regex (setup.py): started
  Building wheel for regex (setup.py): finished with status 'error'
  Running setup.py clean for regex
Failed to build regex
Installing collected packages: regex, autosubmit-api
    Running setup.py install for regex: started
    Running setup.py install for regex: finished with status 'error'

the error on regex:

    regex_3/_regex.c:26230:16: note: declared here
     PyMODINIT_FUNC PyInit__regex(void) {
                    ^~~~~~~~~~~~~
    regex_3/_regex.c:26299:9: warning: implicit declaration of function ‘PyModule_Create’; did you mean ‘PyModule_Check’? [-Wimplicit-function-declaration]
         m = PyModule_Create(&regex_module);
             ^~~~~~~~~~~~~~~
             PyModule_Check
    regex_3/_regex.c:26299:7: warning: assignment to ‘PyObject *’ {aka ‘struct _object *’} from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
         m = PyModule_Create(&regex_module);
           ^
    regex_3/_regex.c:26301:16: warning: ‘return’ with a value, in function returning void
             return NULL;
                    ^~~~
    regex_3/_regex.c:26230:16: note: declared here
     PyMODINIT_FUNC PyInit__regex(void) {
                    ^~~~~~~~~~~~~
    regex_3/_regex.c:26326:16: warning: ‘return’ with a value, in function returning void
             return NULL;
                    ^~~~
    regex_3/_regex.c:26230:16: note: declared here
     PyMODINIT_FUNC PyInit__regex(void) {
                    ^~~~~~~~~~~~~
    regex_3/_regex.c:26329:12: warning: ‘return’ with a value, in function returning void
         return m;
                ^
    regex_3/_regex.c:26230:16: note: declared here
     PyMODINIT_FUNC PyInit__regex(void) {
                    ^~~~~~~~~~~~~
    regex_3/_regex.c: At top level:
    regex_3/_regex.c:26217:27: error: storage size of ‘regex_module’ isn’t known
     static struct PyModuleDef regex_module = {
                               ^~~~~~~~~~~~
    regex_3/_regex.c: In function ‘bytes2_char_at’:
    regex_3/_regex.c:771:1: warning: control reaches end of non-void function [-Wreturn-type]
     }
     ^
    regex_3/_regex.c: In function ‘bytes1_char_at’:
    regex_3/_regex.c:756:1: warning: control reaches end of non-void function [-Wreturn-type]
     }
     ^
    regex_3/_regex.c: In function ‘bytes2_point_to’:
    regex_3/_regex.c:781:1: warning: control reaches end of non-void function [-Wreturn-type]
     }
     ^
    regex_3/_regex.c: In function ‘bytes1_point_to’:
    regex_3/_regex.c:766:1: warning: control reaches end of non-void function [-Wreturn-type]
     }
     ^
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /var/www/html/autosubmitapi/venv27/bin/python2.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-Il9SCI/regex/setup.py'"'"'; __file__='"'"'/tmp/pip-install-Il9SCI/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-sVaToF/install-record.txt --single-version-externally-managed --compile --install-headers /var/www/html/autosubmitapi/venv27/include/site/python2.7/regex Check the logs for full command output.

@wuruchi any insight ? it may be the version of gcc ? the one used was:

gcc version 8.5.0 20210514 (Red Hat 8.5.0-4) (GCC)

In GitLab by @wuruchi on Apr 8, 2022, 18:55

Hello @jberlin

Seems to be a lack of dependency error.

Take a look at this: https://stackoverflow.com/questions/53744668/failed-building-wheel-for-regex-while-installing-pip-package

In GitLab by @wuruchi on Apr 11, 2022, 11:54

Hello @jberlin @mcastril

I've added the Authentication Module page to the wikis of the API and GUI. In this page I explain how the authentication process interacts with each project.

In GitLab by @jberlin on Apr 11, 2022, 12:01

I am still dealing with Regex issue, pip is trying to install a not supported version of this module, the last supported version for old python is 2021.11.10 (based on what is documented here ), which version do you have in your local env @wuruchi ?

In GitLab by @jberlin on Apr 11, 2022, 12:03

previous version worked!

(venv27) [root@vm ~]# pip  --no-cache-dir install regex==2021.11.10
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting regex==2021.11.10
  Downloading regex-2021.11.10.tar.gz (702 kB)
     |████████████████████████████████| 702 kB 1.4 MB/s 
Building wheels for collected packages: regex
  Building wheel for regex (setup.py) ... done
  Created wheel for regex: filename=regex-2021.11.10-cp27-cp27mu-linux_x86_64.whl size=595558 sha256=8d1889d781a314ff5ef2813d1eeab762a467df46a02e2779fe6cee2eeef567a9
  Stored in directory: /tmp/pip-ephem-wheel-cache-IgAO6u/wheels/49/b2/f8/e09563987c2d5d933ee3e652e96183f83c7d2fd34e4f7c1ae1
Successfully built regex
Installing collected packages: regex
Successfully installed regex-2021.11.10

In GitLab by @wuruchi on Apr 11, 2022, 12:05

Hello @jberlin

Great. Just in case, here's a copy of the pip freeze of the virtual environment:

altgraph==0.16.1
aniso8601==7.0.0
apache-libcloud==2.5.0
asn1crypto==0.24.0
autosubmit-api==1.0.17
autosubmitAPIwu==2.9.180
backports.functools-lru-cache==1.5
backports.ssl-match-hostname==3.7.0.1
bcrypt==3.1.7
bleach==3.1.0
bscearth.utils==0.5.2
caniusepython3==7.2.0
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
cheroot==6.5.6
CherryPy==17.4.2
Click==7.0
colorama==0.4.1
contextlib2==0.5.5
cryptography==2.7
cycler==0.10.0
decorator==4.4.0
dis3==0.1.3
distlib==0.3.0
docutils==0.15.2
ecdsa==0.13.2
enum34==1.1.6
Flask==1.1.1
Flask-Cors==3.0.8
Flask-Jsonpify==1.5.0
Flask-RESTful==0.3.7
funcsigs==1.0.2
future==0.17.1
futures==3.3.0
graphviz==0.13
gunicorn==19.9.0
idna==2.8
ipaddress==1.0.22
itsdangerous==1.1.0
jaraco.functools==2.0
Jinja2==2.10.1
kiwisolver==1.1.0
lxml==4.6.3
MarkupSafe==1.1.1
matplotlib==2.2.4
meld3==2.0.0
mock==3.0.5
more-itertools==5.0.0
msgpack==0.6.1
netifaces==0.10.9
networkx==2.2
numpy==1.16.4
packaging==19.2
paramiko==1.15.0
parse==1.12.0
pkginfo==1.5.0.1
portalocker==0.5.7
portend==2.5
pycparser==2.19
pycrypto==2.6.1
pydot==1.4.1
pydotplus==2.0.2
Pygments==2.4.2
PyInstaller==3.5
PyJWT==1.7.1
pymongo==3.8.0
PyNaCl==1.3.0
pyparsing==2.4.0
python-cas==1.5.0
python-dateutil==2.8.0
pytz==2019.1
pyzmq==18.0.2
radical.saga==0.70.0
radical.utils==0.70.0
readme-renderer==24.0
regex==2019.6.8
requests==2.22.0
requests-toolbelt==0.9.1
saga-python==0.60.1
scipy==1.2.2
setproctitle==1.1.10
six==1.12.0
SQLAlchemy==1.3.6
style==1.1.0
subprocess32==3.5.4
supervisor==4.0.4
tempora==1.14.1
tqdm==4.35.0
twine==1.14.0
typing==3.7.4.3
update==0.0.1
urllib3==1.25.3
webencodings==0.5.1
Werkzeug==0.15.5
zc.lockfile==2.0

Most of these packages are no relevant.

In GitLab by @jberlin on Apr 11, 2022, 12:10

now pip installer went through successfully

[root@vm development]# source  /var/www/html/autosubmitapi/venv27/bin/activate
(venv27) [root@vm development]# pip install autosubmit_api
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting autosubmit_api
  Using cached autosubmit_api-1.0.17-py2-none-any.whl (357 kB)
Requirement already satisfied: regex in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (2021.11.10)
Requirement already satisfied: portalocker==0.5.7 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (0.5.7)
Requirement already satisfied: pydotplus==2.0.2 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (2.0.2)
Requirement already satisfied: gunicorn==19.9.0 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (19.9.0)
Requirement already satisfied: Flask-RESTful==0.3.7 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (0.3.7)
Requirement already satisfied: Flask==1.1.1 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (1.1.1)
Requirement already satisfied: Flask-Cors==3.0.8 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (3.0.8)
Requirement already satisfied: bscearth.utils==0.5.2 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (0.5.2)
Requirement already satisfied: requests==2.22.0 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (2.22.0)
Requirement already satisfied: numpy in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (1.16.6)
Requirement already satisfied: paramiko==1.15.0 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (1.15.0)
Requirement already satisfied: networkx==2.2 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (2.2)
Requirement already satisfied: graphviz==0.13 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (0.13)
Requirement already satisfied: Flask-Jsonpify==1.5.0 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (1.5.0)
Requirement already satisfied: mock==3.0.5 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from autosubmit_api) (3.0.5)
Requirement already satisfied: pyparsing>=2.0.1 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from pydotplus==2.0.2->autosubmit_api) (2.4.7)
Requirement already satisfied: six>=1.3.0 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from Flask-RESTful==0.3.7->autosubmit_api) (1.16.0)
Requirement already satisfied: pytz in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from Flask-RESTful==0.3.7->autosubmit_api) (2022.1)
Requirement already satisfied: aniso8601>=0.82 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from Flask-RESTful==0.3.7->autosubmit_api) (9.0.1)
Requirement already satisfied: Werkzeug>=0.15 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from Flask==1.1.1->autosubmit_api) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from Flask==1.1.1->autosubmit_api) (1.1.0)
Requirement already satisfied: click>=5.1 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from Flask==1.1.1->autosubmit_api) (7.1.2)
Requirement already satisfied: Jinja2>=2.10.1 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from Flask==1.1.1->autosubmit_api) (2.11.3)
Requirement already satisfied: python-dateutil in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from bscearth.utils==0.5.2->autosubmit_api) (2.8.2)
Requirement already satisfied: idna<2.9,>=2.5 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from requests==2.22.0->autosubmit_api) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from requests==2.22.0->autosubmit_api) (1.25.11)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from requests==2.22.0->autosubmit_api) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from requests==2.22.0->autosubmit_api) (2021.10.8)
Requirement already satisfied: pycrypto!=2.4,>=2.1 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from paramiko==1.15.0->autosubmit_api) (2.6.1)
Requirement already satisfied: ecdsa>=0.11 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from paramiko==1.15.0->autosubmit_api) (0.17.0)
Requirement already satisfied: decorator>=4.3.0 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from networkx==2.2->autosubmit_api) (4.4.2)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from mock==3.0.5->autosubmit_api) (1.0.2)
Requirement already satisfied: MarkupSafe>=0.23 in /var/www/html/autosubmitapi/venv27/lib/python2.7/site-packages (from Jinja2>=2.10.1->Flask==1.1.1->autosubmit_api) (1.1.1)
Installing collected packages: autosubmit-api
Successfully installed autosubmit-api-1.0.17

@wuruchi can you set the pip installer to use an specific version of regex module, since the installer was trying to use the latest one that is not supported anymore

In GitLab by @wuruchi on Apr 11, 2022, 12:13

@jberlin Sure, I'll add that specific version to the setup.py.

In GitLab by @wuruchi on Apr 11, 2022, 12:15

Done.

In GitLab by @jberlin on Apr 11, 2022, 12:29

I am with the step 3 of the GUI deployment:

3. The API requires an environment variable calledSECRET_KEYto generate tokens for authenticated users. Make sure this variable is set and its value doesn't change too often because that would invalidate tokens faster than necessary.

@wuruchi which value should it have ? which is the correct value for not invalidating tokens too often ?

In GitLab by @wuruchi on Apr 11, 2022, 12:34

Hello @jberlin

A SECRET KEY used to generate tokens could be any value. Usually, you would use a long sequence of characters that is hard to guess.

JWT uses the secret key along with the selected algorithm to encode information in the token. Using that secret key and the same algorithm, it can be decoded. This is an example of symmetric encryption.

In GitLab by @jberlin on Apr 11, 2022, 12:56

Hi @kserrade , it seems that from the VM we will need access to esarchive file system

(venv27) [root@vm ~]# ll /esarchive/autosubmit
ls: cannot access '/esarchive/autosubmit': No such file or directory

Operations is in charge of mouting external file systems ?

In GitLab by @mcastril on Apr 11, 2022, 13:21

@jberlin we cannot access /esarchive for security.

You can copy a few experiments from /esarchive/autosubmit to a local directory in the VM. You don't need to copy the proj so they should not take up much space.

There is also dummy data in https://earth.bsc.es/gitlab/es/autosubmitreact/-/tree/master/data but this is already in the form that the GUI would expect it and it's not for using it with the API right @wuruchi ?

BSC-ES / autosubmit-gui

Test deployment of the API and GUI on the new VM #9