search

c-scale-community / use-case-aquamonitor

Apache License 2.0
2 stars 1 forks source link

Set up compute and data resources to support the VO #4

Closed backeb closed 3 years ago

backeb commented 3 years ago

Set up compute and data resources to support the VO

backeb commented 3 years ago

@gdonvito, @jorge-lip @jopina please start arranging compute and data resources to support the VO. The VO (https://operations-portal.egi.eu/vo/update/serial/841) has been enabled in Perun, see #1 @enolfc is coordinating the SLA, see #2

backeb commented 3 years ago

@gdonvito, @jorge-lip, @jopina please respond :-)

Note that @enolfc suggested we can go ahead with allocating resources to this use case while they set up the SLA https://github.com/c-scale-community/use-case-aquamonitor/issues/2#issuecomment-841284728

mariojmdavid commented 3 years ago

need to know the amount of VA to be provided by INCD for c-scale

backeb commented 3 years ago

@enolfc @sustr4, please assist @mariojmdavid with this.

Below is the information I have from the C-SCALE proposal image

mariojmdavid commented 3 years ago

hi all, I found it in the proposal, and the number of VCPUs is different from those values 4500 VCPU days is wrong in the proposal since it is per month -> 180 VCPUs for the remaining period of the project 450 TB month - 18 TB for the remaining period of the project we will setup the quotas accordingly (this comment was edited to verify the numbers, there is an error in the google sheets and proposal that state this per day instead of per month)

mariojmdavid commented 3 years ago

need to know for the openstack mapping the following about the VO name "any_one_of": ["^urn:mace:egi.eu:group::role=vm_operator#aai.egi.eu$"]

enolfc commented 3 years ago

@mariojmdavid we need to test this as with Perun the entitlements change a bit from the defaults that we use for other VOs.

@backeb have you received my request to be member of the VO?

mariojmdavid commented 3 years ago

@enolfc what is the entitlement? I see that the name of the VO is aquamonitor.c-scale.eu I will create the group, and can do an initial mapping based on that, and we will see later on

mariojmdavid commented 3 years ago

I have configures the usual EGI Checkin entitlement, but you can try the following

you will need to remove the duplicate part "?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/" The is, you should have

https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

reload the page if needed

is a log standing issue in our Openstack deployment, that we are still trying to figure out, but after that you should be able to access the Dashboard

using the keystone CLI with federated identity and OpenID connect tokens should work as expected

backeb commented 3 years ago

@backeb have you received my request to be member of the VO?

@enolfc I don't think so - from which email address did you send it?

backeb commented 3 years ago

@mariojmdavid when I follow the link https://stratus.ncg.ingrid.pt/ and select "EGI Checkin" for "Authenticate using", I get {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

enolfc commented 3 years ago

you should be notified by Perun, not sure to which address though

mariojmdavid commented 3 years ago

@mariojmdavid when I follow the link https://stratus.ncg.ingrid.pt/ and select "EGI Checkin" for "Authenticate using", I get {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

can you confirm (or not) if it's because of my previous comment here? about the duplication in the URL? if yes, check with https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

mariojmdavid commented 3 years ago

if not put a date time when you tried so I can check the logs, or it can be because of the mapping/entitlement

backeb commented 3 years ago

you should be notified by Perun, not sure to which address though

I did get an email from perun, but same problem

backeb commented 3 years ago

@mariojmdavid when I follow the link https://stratus.ncg.ingrid.pt/ and select "EGI Checkin" for "Authenticate using", I get {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

can you confirm (or not) if it's because of my previous comment here? about the duplication in the URL? if yes, check with https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

Hi @mariojmdavid, when I click on the above link (https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/) and log in using EGI SSO I still get the error message: {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}}

I did this now (19 May, 15h12 CET)

mariojmdavid commented 3 years ago

@backeb can you try again ? I had put the "role=vm_operator" now is =member, although some "higher role" should be defined at a certain moment

backeb commented 3 years ago

@backeb can you try again ? I had put the "role=vm_operator" now is =member, although some "higher role" should be defined at a certain moment

@mariojmdavid I get the same error and also tried in "Incognito" on Chrome

mariojmdavid commented 3 years ago

give me an approximate timestamp of this error @enolfc need to know about the mapping string (entitlement) if need to modify because of Perun

backeb commented 3 years ago

give me an approximate timestamp of this error @enolfc need to know about the mapping string (entitlement) if need to modify because of Perun

Time stamp is about the same time as this comment: https://github.com/c-scale-community/use-case-aquamonitor/issues/4#issuecomment-844190140

enolfc commented 3 years ago

@mariojmdavid the current entitlement that users are getting is: urn:mace:egi.eu:group:aquamonitor:members:role=member#aai.egi.eu

We many add more restrictive groups in Perun if needed

mariojmdavid commented 3 years ago

entitlement updated @backeb can you try again?

enolfc commented 3 years ago

@mariojmdavid I confirm it works, I managed to get access to the VO.

backeb commented 3 years ago

Hi @gena,

@mariojmdavid @gdonvito @enolfc need to know how much storage resources they need to provide for Aquamonitor.

The list of data to be provided for Aquamonitor can be accessed here, I copy-pasted the list below: image

Could you (@gena) give an indication of how much storage you would need?

Thanks

backeb commented 3 years ago

entitlement updated @backeb can you try again?

@mariojmdavid I can confirm that when I try with the below link I can get access to the OpenStack dashboard: https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

Will you set up an instance for us? Once @gena has confirmed the amount of storage needed?

gena commented 3 years ago

For Aqua Monitor, if we will try reproducing global analysis at 30m - something like 5TB should be enough. If the analysis will be limited to specific areas - probably 1TB would be ok.

Are there options to increase storage when needed? Or it is possible to do only by mounting additional disk and migrating datasets?

enolfc commented 3 years ago

entitlement updated @backeb can you try again?

@mariojmdavid I can confirm that when I try with the below link I can get access to the OpenStack dashboard: https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/

Will you set up an instance for us? Once @gena has confirmed the amount of storage needed?

I would assume you can do the setup of the instance yourselves. If you need support we can guide you through it, but I'd prefer that we do not start VMs for users, as users need be responsible of their resources.

enolfc commented 3 years ago

For Aqua Monitor, if we will try reproducing global analysis at 30m - something like 5TB should be enough. If the analysis will be limited to specific areas - probably 1TB would be ok.

Are there options to increase storage when needed? Or it is possible to do only by mounting additional disk and migrating datasets?

@gena, the VMs can have volumes attached to them and these volumes are quite flexible for growing (shrinking not so easy). Additional disks can also be setup if needed.

backeb commented 3 years ago

entitlement updated @backeb can you try again?

@mariojmdavid I can confirm that when I try with the below link I can get access to the OpenStack dashboard: https://stratus.ncg.ingrid.pt:5000/v3//auth/OS-FEDERATION/websso/openid?origin=https://stratus.ncg.ingrid.pt/dashboard/auth/websso/ Will you set up an instance for us? Once @gena has confirmed the amount of storage needed?

I would assume you can do the setup of the instance yourselves. If you need support we can guide you through it, but I'd prefer that we do not start VMs for users, as users need be responsible of their resources.

Thanks @enolfc. I assume the use case leads should set up the instances (I can do this for Deltares).

Once I have created the instance, do I just add the public keys of the developers to the VM so they can access it via SSH? Or do I also have to add them in PERUN? Could you provide me with some idea of the next steps?

enolfc commented 3 years ago

Once I have created the instance, do I just add the public keys of the developers to the VM so they can access it via SSH? Or do I also have to add them in PERUN? Could you provide me with some idea of the next steps?

Adding the ssh keys is good enough to get started. We could check how to automate this with PERUN if you think this is interesting (as with every automation it depends on how many times you need to do this)

backeb commented 3 years ago

Hi @gena and @avgils

I now have access to INCD’s OpenStack environment (https://stratus.ncg.ingrid.pt/dashboard/project/). To access the environment you need to

@avgils on Monday let us sit together and create an instance.

mariojmdavid commented 3 years ago

hi all you should do the instantiation of the VMs as you need, apologies but I will not do it :D for the storage you should plan as much as you need, you can go up to 18TB which is the quota for the whole VO and of course I will be here to help and answer questions as you have

backeb commented 3 years ago

hi all you should do the instantiation of the VMs as you need, apologies but I will not do it :D for the storage you should plan as much as you need, you can go up to 18TB which is the quota for the whole VO and of course I will be here to help and answer questions as you have

Hi @mariojmdavid

I've set up an instance image

I've associated two IP addresses to the instance (one public I think). When I try logging in I get the following error:

PS C:\Users\backeber\keys> ssh 194.210.120.109 -i "path\to\pvt_key.pem"
The authenticity of host '194.210.120.109 (194.210.120.109)' can't be established.
ECDSA key fingerprint is SHA256:gC1fqw9gZ67ApTwLs2NtRSrUIZAlYczOFs2/8t/el5U.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '194.210.120.109' (ECDSA) to the list of known hosts.
directory\\backeber@194.210.120.109: Permission denied (publickey).

It seems the public key is rejecting the private key. Any suggestions?

avgils commented 3 years ago

Hi @mariojmdavid

I'm Anna van Gils from Deltares and I should have access to the Aquamonitor recourses as well. I registered my credentials with EGI SSO (https://www.egi.eu/sso). My mail adress is anna.vangils@deltares.nl, username gilsa

Can you link my account so I can access the OpenStack environment?

Regards,

Anna

enolfc commented 3 years ago 
PS C:\Users\backeber\keys> ssh 194.210.120.109 -i "path\to\pvt_key.pem"
The authenticity of host '194.210.120.109 (194.210.120.109)' can't be established.
ECDSA key fingerprint is SHA256:gC1fqw9gZ67ApTwLs2NtRSrUIZAlYczOFs2/8t/el5U.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '194.210.120.109' (ECDSA) to the list of known hosts.
directory\\backeber@194.210.120.109: Permission denied (publickey).

It seems the public key is rejecting the private key. Any suggestions?

What image did you start? Should you try with the default user of the image (e.g. ubuntu)?

enolfc commented 3 years ago

Hi @mariojmdavid

I'm Anna van Gils from Deltares and I should have access to the Aquamonitor recourses as well. I registered my credentials with EGI SSO (https://www.egi.eu/sso). My mail adress is anna.vangils@deltares.nl, username gilsa

Can you link my account so I can access the OpenStack environment?

Regards,

Anna

Hi @avgils, you need to get membership into the VO. this should be the right link: https://perun.egi.eu/gui/registrar/?vo=aquamonitor

backeb commented 3 years ago 
PS C:\Users\backeber\keys> ssh 194.210.120.109 -i "path\to\pvt_key.pem"
The authenticity of host '194.210.120.109 (194.210.120.109)' can't be established.
ECDSA key fingerprint is SHA256:gC1fqw9gZ67ApTwLs2NtRSrUIZAlYczOFs2/8t/el5U.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '194.210.120.109' (ECDSA) to the list of known hosts.
directory\\backeber@194.210.120.109: Permission denied (publickey).

It seems the public key is rejecting the private key. Any suggestions?

What image did you start? Should you try with the default user of the image (e.g. ubuntu)?

@enolfc we used: ubuntu-20.04-amd64

avgils commented 3 years ago

Hi @mariojmdavid I'm Anna van Gils from Deltares and I should have access to the Aquamonitor recourses as well. I registered my credentials with EGI SSO (https://www.egi.eu/sso). My mail adress is anna.vangils@deltares.nl, username gilsa Can you link my account so I can access the OpenStack environment? Regards, Anna

Hi @avgils, you need to get membership into the VO. this should be the right link: https://perun.egi.eu/gui/registrar/?vo=aquamonitor

At the moment I get the following Error:

Screenshot 2021-05-31 at 14 10 47
mariojmdavid commented 3 years ago

@backeb ssh ubuntu@194.210.120.109 ...

backeb commented 3 years ago

Thanks @mariojmdavid, we now have access via: ssh -i "path/to/private_key.pem" ubuntu@194.210.120.109

backeb commented 3 years ago

Hi @mariojmdavid I'm Anna van Gils from Deltares and I should have access to the Aquamonitor recourses as well. I registered my credentials with EGI SSO (https://www.egi.eu/sso). My mail adress is anna.vangils@deltares.nl, username gilsa Can you link my account so I can access the OpenStack environment? Regards, Anna

Hi @avgils, you need to get membership into the VO. this should be the right link: https://perun.egi.eu/gui/registrar/?vo=aquamonitor

At the moment I get the following Error:

Screenshot 2021-05-31 at 14 10 47

@melanger could you please advise regarding this comment, please: https://github.com/c-scale-community/use-case-aquamonitor/issues/4#issuecomment-851448071

maricaantonacci commented 3 years ago

Dear @backeb , @enolfc we have finalized the setup of the aquamonitor VO at our site (INFN-CLOUD-BARI). Openstack dashboard is accessible at cloud.recas.ba.infn.it, choose OpenID Connect Authentication and then click on aai.egi.eu/oidc/ link. If everything is working fine you should land in the project C-SCALE_aquamonitor. Please let us know in case of problems Cheers

mariojmdavid commented 3 years ago

hi all I think this issue should be closed as both resource providers have setup the VO and resources any further possible problems should be followed in other issues

backeb commented 3 years ago

Agreed @mariojmdavid. The final action here is on @backeb and @avgils to instantiate a VM on INFN-CLOUD-BARI's OpenStack dashboard.