search

c-scale-community / workflow-coastal-hydrowaq

Porting and deploying the HiSea use case on C-SCALE
Apache License 2.0
3 stars 1 forks source link

Hybrid cloud - HPC workflow #40

Open nikosT opened 1 year ago

nikosT commented 1 year ago

Flow Script

Oriented for the HPC part, of course

This can be viewed in the flow branch: https://github.com/c-scale-community/use-case-hisea/tree/flow

FYI @backeb @sebastian-luna-valero @lorincmeszaros

(please resolve it when you check it)

sebastian-luna-valero commented 1 year ago

Thanks @nikosT

It looks like a great starting point!

backeb commented 1 year ago

I will take a look, probably later this month / early next month.

Thanks so much for sharing

backeb commented 1 year ago

I've requested access to ARIS. Sent the below to support@hpc.grnet.gr

Hi there,

I have just requested to join the HiSea CO: https://sram.surf.nl/registration?collaboration=9b0a6184-326e-44d8-bc1d-903193b11f2b

Once approved I will upload my public key to my SRAM profile.

The IP address I am connecting from is: 192.168.178.72

Please advise how to access aris.

Thanks Björn

backeb commented 1 year ago

Request approved. Now waiting for feedback from support@hpc.grnet.gr on how to access.

sebastian-luna-valero commented 1 year ago

Hi @backeb

The IP address I am connecting from is: 192.168.178.72

This looks like a private IP. Try accessing something like https://ip.me/ on your web browser to get the public IP you are using.

Note, if this is a dynamic address it will change from time to time. To solve this I asked support@hpc.grnet.gr to configure access from https://notebooks.egi.eu/ so after uploading your SSH key to SRAM and getting logging instructions from support@hpc.grnet.gr you can go to the EGI notebooks, open a terminal and ssh into ARIS.

I hope it helps!

backeb commented 1 year ago

Thanks @sebastian-luna-valero

The IP I get from https://ip.me/ is 2a02:a210:548:4480:b525:aa5c:4159:8fc9.

Perhaps though it would be better to access ARIS from the GRNET OpenStack VM we're working on... since we're trying to connect to the workflow from there.

The ifconfig output on the GRNET OpenStack VM is: 

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:11ff:fe2c:32e4  prefixlen 64  scopeid 0x20<link>
        ether 02:42:11:2c:32:e4  txqueuelen 0  (Ethernet)
        RX packets 1081736  bytes 146873375 (140.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1642610  bytes 19185023358 (17.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 8950
        inet 192.168.0.124  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::f816:3eff:fe72:4220  prefixlen 64  scopeid 0x20<link>
        ether fa:16:3e:72:42:20  txqueuelen 1000  (Ethernet)
        RX packets 16111980  bytes 39437808757 (36.7 GiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 15672067  bytes 53572994368 (49.8 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 357084  bytes 43303191 (41.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 357084  bytes 43303191 (41.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
sebastian-luna-valero commented 1 year ago

Very good point, in that case the correct IP (from OpenStack dashboard) is: 62.217.83.202

backeb commented 1 year ago

@nikosT, I think the action is on you to explain access to ARIS ;-)

cc @lorincmeszaros

nikosT commented 1 year ago

Hi @backeb and @sebastian-luna-valero, as far as I can see (by e-mail exchanges) you currently have access. Correct?

backeb commented 1 year ago

I think so, will continue on Friday.

backeb commented 1 year ago

@nikosT I received the below:

Hello, Your account is ready. You may login with ssh from your IPs, using: ssh bbackeberg2@login03.aris.grnet.gr Your slurm accounting project (the option #SBATCH -A or #SBATCH --account) is hisea. Jobs should be submitted to the el7thin or el7taskp partitions (the option #SBATCH -p or #SBATCH --partition). Detailed ARIS documentation can be found at http://doc.aris.grnet.gr/ If you have any questions, technical issues etc. please email support@hpc.grnet.gr Regards, Kyriakos Gkinis GRNET

When I log on via ssh it asks for a password, what's the password?

sebastian-luna-valero commented 1 year ago

Some checks:

  1. Have you uploaded your public SSH key to https://sram.surf.nl/profile ?
  2. Are you using ssh -i ~/.ssh/id_rsa -l bbackeberg2 login03.aris.grnet.gr where ~/.ssh/id_rsa is replaced with the path to your private SSH key? This private key should be the pair of the public SSH key uploaded to SRAM in the previous step.
backeb commented 1 year ago

I've done the following tests:

  1. my id_rsa.pub is uploaded to https://sram.surf.nl/profile. In the OpenStack VM, I appended my id_rsa.pub to /home/centos/.ssh/id_rsa.pub and tried ssh -i ~/.ssh/id_rsa.pub bbackeberg2@login03.aris.grnet.gr ➡️ it still asked for my password.
  2. I then uploaded the OpenStack VM public key from /home/centos/.ssh/id_rsa.pub to https://sram.surf.nl/profile and tried ssh -i ~/.ssh/id_rsa.pub bbackeberg2@login03.aris.grnet.gr ➡️ it still asked for my password.
sebastian-luna-valero commented 1 year ago

Step 2 is the solution. Try this instead:

ssh -i ~/.ssh/id_rsa bbackeberg2@login03.aris.grnet.gr
backeb commented 1 year ago

Step 2 is the solution. Try this instead:

ssh -i ~/.ssh/id_rsa bbackeberg2@login03.aris.grnet.gr

Still asks for password

sebastian-luna-valero commented 1 year ago

If you just uploaded /home/centos/.ssh/id_rsa.pub to https://sram.surf.nl/profile I believe it needs to be synced with the LDAP server at ARIS. Let's see what @nikosT has to say.

backeb commented 1 year ago

I'll also contact support.

backeb commented 1 year ago

Sent the following to support:

Hi there,

When I log on via ssh it asks for a password. I’ve done the following tests:

  1. my id_rsa.pub is uploaded to https://sram.surf.nl/profile. In the OpenStack VM, I appended my id_rsa.pub to /home/centos/.ssh/id_rsa.pub and tried ssh -i ~/.ssh/id_rsa.pub bbackeberg2@login03.aris.grnet.gr ➡️ it still asked for my password.
  2. I then uploaded the OpenStack VM public key from /home/centos/.ssh/id_rsa.pub to https://sram.surf.nl/profile and tried ssh -i ~/.ssh/id_rsa.pub bbackeberg2@login03.aris.grnet.gr ➡️ it still asked for my password.

We think that https://sram.surf.nl/profile needs to be synced with the LDAP server at ARIS.

Please advise.

Thanks Bjorn

lorincmeszaros commented 1 year ago

@sebastian-luna-valero I have another interesting issue. My SRAM profile says that I am not a member of any colllaboration. It is weird because I am visible as a member. In any case, I sent you a request to (re)join the hisea.c-scale collaboration. Pleas accept.

nikosT commented 1 year ago

Hi everyone, the procedure from putting new SSH keys to SRAM until they are activated is not automatic. This means an ARIS admin adds the new SSH-keys manually. I requested from ARIS admins to add the new SSH keys you just put in the SRAM @backeb @lorincmeszaros. Additionally, @lorincmeszaros I see you in the use case. @backeb it should NOT ask for a password. If that is happening check if the SSH keys or the public IP is the one that you provided to us.

backeb commented 1 year ago

For the sake of completion, the SSH public keys I have added in SRAM are:

  1. the OpenStack VM public key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDO4Zjerg2NUMieVrOdwdPvbJm0sQYYDtpqiZJfPWfF3arIcCsb0jopm1Wa6pamUf94czrbhcOZSP4IZ/bISrJWttOaXhM+y26N879Zq5hX748hvuKg/CrtNFSdJBa79jh9fWIsZpgtAx15r8DMXHkM1y/L8fX/B+aNoeqL1rlgQtxEKoYd/lQUQfgz9OM4GDtTbmvgpH/+rF1N0ZRouuhjbzRwf033Tu1T5sgFwGu5a2jyZiD08CLVCcJAu8I0fF6NrSe8hILh7qtAGno1UnDrdjnt9rQ64aKGJgzPzQxvRwvu8yHE8ACuYf5UCf+obBdU4C2LX8Te+BczsRsMT1Q4H2atlZLA/QcjTf/IqqfW2ngiNysufOHGW9btqdkEEy9XbU9+NN2uu43WqJOm7kgX9Fn3kgyz4rhiBXFJeVahvQ8CESS8zJua+NFdMpLb5rHTeuZ/s6hjXmQPbVofw2cSOWYRA+QmQGI+2a61ww3CoHWwtEul5BOKkVjMs7fB2S020dOQ0JgzrRqzEO7FDkxwlJ7usBtRgJwYHVs8S3oZeHVWfWmI2s50weZ2kMrwCaKUMwKUjHIxUtoes0/AjC32CPP/g0W+c/tTF+bMssOzLoE89LhoyHCcerw9Cf6/m5roGy0J1ELZXRB8lYjnFjrZhGMkhGbmzBnNMy0sC+GulQ== centos@hisea-test-vm.os.grnetcloud.net
  2. my public key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzDQt0T/AxObZa56nUs8CexOxVqLf2wiUpbdUxU132O4uNSm6Sf3M7mLyWwkroYO2BlZ6cdZRaqfkEzdjQFQhouTyRxT7AiOcyHOagOhhvbiq7T3bgNgLP24hRnHpuxktLt1Q4U4IOHh2GbnIyNNFWgT4KfI7Nz+OsokvDcz4d9CXNs759rHzL4NYa5B3s XiGrP84NK7L2LhvHO/CX9+2OM5pd8wilCG4HgpIbCvjTtERsvsFiU+Z1xi7KT4M5WEUyDQrS8nVKrsb2+2yRzrQ0A1sPTn5Ih4y31S7KJ5EDmobzjGb9cEqfVjyMDIVVxDUx771bNmKwIQMnmEpWIgwjm0c19fFwkbYq3EN8BrdJc1n9q1ABw0HpTX7yNGUAn2qrsPTl1G3ZVyfzN0aD1XzWqD69ZaEYG4VU nO9skcex5VDgO5ZXN/US/l2NiX/x698YgvY1fYUO8jMRGw3uRox/C4tTQj7ifQIgYwXGUEG2W1nFJp/5EMD5vZNrQojB9Hk= backeb@Bjorns-MacBook-Pro-2.local
backeb commented 1 year ago

Response from support:

We have added your second key (the centos@hisea-test-vm.os.grnetcloud.net one) to your authorized_keys. The first key (the backeb@Bjorns-MacBook-Pro-2.local one) was already there.

You are giving the wrong argument to the ssh -i option, it should be your private key and not your public key.

lorincmeszaros commented 1 year ago

Hi everyone, the procedure from putting new SSH keys to SRAM until they are activated is not automatic. This means an ARIS admin adds the new SSH-keys manually. I requested from ARIS admins to add the new SSH keys you just put in the SRAM @backeb @lorincmeszaros. Additionally, @lorincmeszaros I see you in the use case. @backeb it should NOT ask for a password. If that is happening check if the SSH keys or the public IP is the one that you provided to us.

hi @nikosT , I have just updated my public SSH in SRAM. Could you request the ARIS admins to add this new key. thank you.

nikosT commented 1 year ago

Hi everyone, the procedure from putting new SSH keys to SRAM until they are activated is not automatic. This means an ARIS admin adds the new SSH-keys manually. I requested from ARIS admins to add the new SSH keys you just put in the SRAM @backeb @lorincmeszaros. Additionally, @lorincmeszaros I see you in the use case. @backeb it should NOT ask for a password. If that is happening check if the SSH keys or the public IP is the one that you provided to us.

hi @nikosT , I have just updated my public SSH in SRAM. Could you request the ARIS admins to add this new key. thank you.

Hi @lorincmeszaros , the request has been made, I'll let you know when it is ready. In any case please use your account name of Aris when you try to log in, that is WITHOUT the number "2".

nikosT commented 1 year ago

Hi @lorincmeszaros , both ssh keys have been added to the same account in ARIS. Could you check that you can connect?

lorincmeszaros commented 1 year ago

Hi @lorincmeszaros , both ssh keys have been added to the same account in ARIS. Could you check that you can connect?

Thank you. It works now with my acount name wothout the number "2"!

lorincmeszaros commented 1 year ago

@nikosT Another question. Could I get Midnight Commander on ARIS?

nikosT commented 1 year ago

@nikosT Another question. Could I get Midnight Commander on ARIS?

@lorincmeszaros , could you please request the tool by sending an e-mail to: support@hpc.grnet.gr?

lorincmeszaros commented 1 year ago

support@hpc.grnet.gr

yes, I have sent a request

nikosT commented 1 year ago

support@hpc.grnet.gr

yes, I have sent a request

Hi @lorincmeszaros , the tool is installed.