search

dCache / dcache

dCache - a system for storing and retrieving huge amounts of data, distributed among a large number of heterogenous server nodes, under a single virtual filesystem tree with a variety of standard access methods
https://dcache.org
285 stars 136 forks source link

Is it possible to define a wlcg-tape-rest-api.path for the webdav door? #7185

Closed elenamplanas closed 1 year ago

elenamplanas commented 1 year ago

We've different VOs and each one should have their own wlcg-tape-rest-api.json file.

The frontend.wellknown!wlcg-tape-rest-api.path option allows us to point correctly to the correct file, but Rucio is redirected to our webdav port endpoint, and we would like to change this default.

If it's very complicated, don't hesitate. We'll try to modify the Rucio SE.

Thanks!

alrossi commented 1 year ago

Hi Elena,

sorry for the delay ... my vacation got extended by a week.

Be aware that the .well-known is exported on both 2880 and 3880.

I'm not sure why you need different endpoints for different VOs. Could you explain this use case a little more?

Thanks, Al

elenamplanas commented 1 year ago

Hi Al, We've different webdav doors and it's own frontend because there are different login requirements and root paths. This allows us to balance load in different physical doors.

We suspect that the wellknown functionality would be used only by LHC experiments, and for the moment, those webdav doors are the ones with more load and thus they are in separate doors. Then, in each door there is the file with the config for that VO, and it's working as expected.

Other external experiments when use the TAPE REST, they know the port and don't have to query the wellknown information.

Do you recommend to have and unique endpoint for all TAPE REST requests? We already have different srm for each LHC VOs, and a generic one for other requests.

Cheers, Elena

alrossi commented 1 year ago

Hi Elena,

So if you are simply running different webdav/frontends, there is no reason you can't set the property to a separate file for each one ... unless I'm misunderstanding something here.

Example:

[frontend-VO-1Domain]
frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-1.json
[frontend-VO-1Domain/frontend]
[frontend-VO-1Domain/webdav]

[frontend-VO-2Domain]
frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-2.json
[frontend-VO-2Domain/frontend]
[frontend-VO-2Domain/webdav]

with the two separate files pointing to the appropriate frontend urls/ports.

But I trust you have already figured this out? Or is there still an issue for you?

Let me know,

Al

elenamplanas commented 1 year ago

Hi Al,

Yes, I've tested it and I can setup different files if the query to the wellknown is done through the frontend, but the question is that they are asking (for example ATLAS) using the SE endpoint webdav defined for rucio , which is the one they use for http or https transfers. In this case there is no a parameter to define a different file,

Don't hesitate, we've set a workaround and its running fine with this configuration.

Thanks!

alrossi commented 1 year ago

Now I'm a bit confused again.

Whatever port webdav is running on, the frontend property should be loaded there. There is no separate webdav property because the frontend property for the well-known file applies to both.

Perhaps this is my confusion?: You want the .well-known webdav TAPE API to be different from the frontend .well-known? That is, you don't want to pair frontend and webdav services?

I would have to change the factory code to look for separate properties in that case. It could be done, if that is what you need.

Perhaps you wouldn't mind sharing the configuration with me so I can fully grasp this ... if you want to do it privately, we can shift this discussion to an RT ticket.

alrossi commented 1 year ago

Let me add also:

[webDavDomain]
frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-1.json
[webDavDomain/webdav]
[frontendDomain]
frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-2.json
[frontendDomain/frontend]

is also entirely possible, though it looks peculiar. (There has been a recent pull-request to our documentation to clarify this fact that webDav is using frontend properties).

Perhaps the solution is to remove the frontend-specific property and relocate it to dcache.properties as

dcache.wellknown!wlcg-tape-rest-api.path

...

elenamplanas commented 1 year ago

Hi Al,

sorry for all the noise...

I had set the frontend.wellknown!wlcg-tape-rest-api.path into the cell definition. The "frontend" name confused me and I supposed that it only applies to frontend.

I've done a new test setting the parameter for the domain, and now it's working perfectly.

Thanks for everything!

Cheers, Elena

Missatge de Albert Rossi @.***> del dia dj., 22 de juny 2023 a les 17:14:

Let me add also:

[webDavDomain] frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-1.json [webDavDomain/webdav]

[frontendDomain] frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-2.json [frontendDomain/frontend]

is also entirely possible, though it looks peculiar. (There has been a recent pull-request to our documentation to clarify this fact that webDav is using frontend properties).

Perhaps the solution is to remove the frontend-specific property and relocate it to dcache.properties as

dcache.wellknown!wlcg-tape-rest-api.path

...

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/7185#issuecomment-1602820430, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGJO2427H3MXGY5VZQFPWSTXMRON7ANCNFSM6AAAAAAZAPM5OI . You are receiving this because you authored the thread.Message ID: @.***>

-- Elena Planas PIC/Institut de Física d’Altes Energies (IFAE), The Barcelona Institute of Science and Technology, Campus UAB 08193 Bellaterra (Barcelona) +34-93 164 16 55

alrossi commented 1 year ago

Hi Elena,

No problem. The burden of clarification rests on our (my) shoulders here. Since two users now have had this problem, I will definitely do something to make this look more like dCache practice.

I assume I can close this issue?

Cheers, Al

Albert L. Rossi Senior Software Developer Scientific Computing Division, Scientific Data Services, Distributed Data Development WH 566 Fermi National Accelerator Laboratory Batavia, IL 60510 (630) 840-3023

From: elenamplanas @.> Sent: Friday, June 23, 2023 6:55 AM To: dCache/dcache @.> Cc: Albert Rossi @.>; Comment @.> Subject: Re: [dCache/dcache] Is it possible to define a wlcg-tape-rest-api.path for the webdav door? (Issue #7185)

Hi Al,

sorry for all the noise...

I had set the frontend.wellknown!wlcg-tape-rest-api.path into the cell definition. The "frontend" name confused me and I supposed that it only applies to frontend.

I've done a new test setting the parameter for the domain, and now it's working perfectly.

Thanks for everything!

Cheers, Elena

Missatge de Albert Rossi @.***> del dia dj., 22 de juny 2023 a les 17:14:

Let me add also:

[webDavDomain] frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-1.json [webDavDomain/webdav]

[frontendDomain] frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-2.json [frontendDomain/frontend]

is also entirely possible, though it looks peculiar. (There has been a recent pull-request to our documentation to clarify this fact that webDav is using frontend properties).

Perhaps the solution is to remove the frontend-specific property and relocate it to dcache.properties as

dcache.wellknown!wlcg-tape-rest-api.path

...

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/7185#issuecomment-1602820430https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_issues_7185-23issuecomment-2D1602820430-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=y0WQkpmhs4d4CKOyCfoQse3BU9UMx42BNsHVQudhbFHGcuJ31CjwGxVQk62xLedR&s=yqAX61Ss8D_cxj7VrCimTyeN3A5qF2GcQwIcFTjfP3w&e=, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGJO2427H3MXGY5VZQFPWSTXMRON7ANCNFSM6AAAAAAZAPM5OIhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AGJO2427H3MXGY5VZQFPWSTXMRON7ANCNFSM6AAAAAAZAPM5OI-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=y0WQkpmhs4d4CKOyCfoQse3BU9UMx42BNsHVQudhbFHGcuJ31CjwGxVQk62xLedR&s=myHY-Z7JNU4RrNJeycJ5S9Bv7Q6ole35pQNbiZZFWh0&e= . You are receiving this because you authored the thread.Message ID: @.***>

-- Elena Planas PIC/Institut de Física d’Altes Energies (IFAE), The Barcelona Institute of Science and Technology, Campus UAB 08193 Bellaterra (Barcelona) +34-93 164 16 55

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_issues_7185-23issuecomment-2D1604175882&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=y0WQkpmhs4d4CKOyCfoQse3BU9UMx42BNsHVQudhbFHGcuJ31CjwGxVQk62xLedR&s=Wa16N3LL_FN3-IdF95Zc3_wgvdmsjVi3xLBX_SWI7NA&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA6NBHGRTZGPWYLYJOZU4JLXMV7Z5ANCNFSM6AAAAAAZAPM5OI&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=y0WQkpmhs4d4CKOyCfoQse3BU9UMx42BNsHVQudhbFHGcuJ31CjwGxVQk62xLedR&s=5FoH5mGnN9GSNq441MKMcu2FbIn46_qIB271HEOGE84&e=. You are receiving this because you commented.Message ID: @.***>

elenamplanas commented 1 year ago

Sure! Thanks for everything!

Cheers, Elena

Missatge de Albert Rossi @.***> del dia dv., 23 de juny 2023 a les 14:18:

Hi Elena,

No problem. The burden of clarification rests on our (my) shoulders here. Since two users now have had this problem, I will definitely do something to make this look more like dCache practice.

I assume I can close this ticket?

Cheers, Al

Albert L. Rossi Senior Software Developer Scientific Computing Division, Scientific Data Services, Distributed Data Development WH 566 Fermi National Accelerator Laboratory Batavia, IL 60510 (630) 840-3023

From: elenamplanas @.> Sent: Friday, June 23, 2023 6:55 AM To: dCache/dcache @.> Cc: Albert Rossi @.>; Comment @.> Subject: Re: [dCache/dcache] Is it possible to define a wlcg-tape-rest-api.path for the webdav door? (Issue #7185)

Hi Al,

sorry for all the noise...

I had set the frontend.wellknown!wlcg-tape-rest-api.path into the cell definition. The "frontend" name confused me and I supposed that it only applies to frontend.

I've done a new test setting the parameter for the domain, and now it's working perfectly.

Thanks for everything!

Cheers, Elena

Missatge de Albert Rossi @.***> del dia dj., 22 de juny 2023 a les 17:14:

Let me add also:

[webDavDomain]

frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-1.json

[webDavDomain/webdav]

[frontendDomain]

frontend.wellknown!wlcg-tape-rest-api.path=/var/lib/httpd/wlcg-tape-rest-api-for-vo-2.json

[frontendDomain/frontend]

is also entirely possible, though it looks peculiar. (There has been a recent pull-request to our documentation to clarify this fact that webDav is using frontend properties).

Perhaps the solution is to remove the frontend-specific property and relocate it to dcache.properties as

dcache.wellknown!wlcg-tape-rest-api.path

...

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/7185#issuecomment-1602820430< https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_issues_7185-23issuecomment-2D1602820430-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=y0WQkpmhs4d4CKOyCfoQse3BU9UMx42BNsHVQudhbFHGcuJ31CjwGxVQk62xLedR&s=yqAX61Ss8D_cxj7VrCimTyeN3A5qF2GcQwIcFTjfP3w&e=>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AGJO2427H3MXGY5VZQFPWSTXMRON7ANCNFSM6AAAAAAZAPM5OI < https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AGJO2427H3MXGY5VZQFPWSTXMRON7ANCNFSM6AAAAAAZAPM5OI-253E&d=DwQFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=y0WQkpmhs4d4CKOyCfoQse3BU9UMx42BNsHVQudhbFHGcuJ31CjwGxVQk62xLedR&s=myHY-Z7JNU4RrNJeycJ5S9Bv7Q6ole35pQNbiZZFWh0&e=>

. You are receiving this because you authored the thread.Message ID: @.***>

-- Elena Planas PIC/Institut de Física d’Altes Energies (IFAE), The Barcelona Institute of Science and Technology, Campus UAB 08193 Bellaterra (Barcelona) +34-93 164 16 55

— Reply to this email directly, view it on GitHub< https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_issues_7185-23issuecomment-2D1604175882&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=y0WQkpmhs4d4CKOyCfoQse3BU9UMx42BNsHVQudhbFHGcuJ31CjwGxVQk62xLedR&s=Wa16N3LL_FN3-IdF95Zc3_wgvdmsjVi3xLBX_SWI7NA&e=>, or unsubscribe< https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA6NBHGRTZGPWYLYJOZU4JLXMV7Z5ANCNFSM6AAAAAAZAPM5OI&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=y0WQkpmhs4d4CKOyCfoQse3BU9UMx42BNsHVQudhbFHGcuJ31CjwGxVQk62xLedR&s=5FoH5mGnN9GSNq441MKMcu2FbIn46_qIB271HEOGE84&e=>.

You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/dCache/dcache/issues/7185#issuecomment-1604203058, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGJO24Y5MFRBMVGLNZV6N4TXMWCSZANCNFSM6AAAAAAZAPM5OI . You are receiving this because you authored the thread.Message ID: @.***>

-- Elena Planas PIC/Institut de Física d’Altes Energies (IFAE), The Barcelona Institute of Science and Technology, Campus UAB 08193 Bellaterra (Barcelona) +34-93 164 16 55