search

dmwm / CMSRucio

7 stars 31 forks source link

`rucio list-dataset-replicas` throws HTTP 413 for some datasets #681

Open haozturk opened 11 months ago

haozturk commented 11 months ago

We've been reported that rucio list-dataset-replicas throws HTTP error with status code 413 for a dataset [1]:

$ rucio list-dataset-replicas cms:/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW
2023-12-13 13:52:18,763 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 413)

I suspect, it's due to the fact it has too many files as status code 413 suggest "Content Too Large". This is a minor issue. We can look into it if we observe it for more datasets. I just wanted to have a record of this.

[1] https://its.cern.ch/jira/browse/CMSPROD-126

labkode commented 5 months ago

The request is roughly ~4.5MB (Content-Length: 4526110), see below.

Rucio uses for the server component Apache server, which default body size is unlimited (~ before June 2022) and 1GB afterwards.

Common ingresses are NGINX (default limit 1 MB) and Traefik (unlimited).

So, smells like infrastructure limit. Bumping ingress limits should fix the issue.

sendto(3, "POST /replicas/datasets_bulk HTTP/1.1\r\nHost: cms-rucio.cern.ch\r\nUser-Agent: rucio-clients/32.3.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: Keep-Alive\r\nX-Rucio-Auth-Token: gonzalhu-/DC=ch/DC=cern/OU=Organic
 Units/OU=Users/CN=gonzalhucms/CN=756361/CN=Hugo Gonzalxxx\r\nX-Rucio-VO: def\r\nX-Rucio-Script: rucio::list-dataset-replicas\r\nContent-Length: 4526110\r\n\r\n", 420, 0, NULL, 0) = 420
sendto(3, "{\"dids\": [{\"scope\": \"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW#000083ba-859a-11e6-a887-001e67abef8c\"}, {\"scope\": \"cms\", \"name\": \"
/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW#00009d64-859a-11e6-a887-001e67abef8c\"}, {\"scope\": \"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80
X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW#0007448e-85c7-11e6-b48c-002590494fb0\"}, {\"scope\": \"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RA
W#00349c92-84d0-11e6-a104-002590494c06\"}, {\"scope\": \"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v2-v2/GEN-SIM-DIGI-RAW#0034c050-84d0-11e6-a104-002590494c06\"}, {\"scope\": \
"cms\", \"name\": \"/Neutrino_E-10_gun/RunIISpring15PrePremix-PUMoriond17_80X_mcRun2_asymptotic_2016_Tranche"..., 4526110, 0, NULL, 0) = 4526110
munmap(0x7fc452269000, 4530176)         = 0
ioctl(3, FIONBIO, [0])                  = 0
recvfrom(3, "HTTP/1.1 413 Request Entity Too Large\r\nDate: Thu, 23 May 2024 14:23:56 GMT\r\nContent-Type: text/html\r\nContent-Length: 176\r\nConnection: close\r\n\r\n<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n", 8192, 0, NULL, NULL) = 319
close(3)