pangeo-data / zarr-proxy

A proxy for Zarr stores that allows for chunking overrides.
Apache License 2.0
9 stars 3 forks source link

add validation for max chunk size #14

Closed andersy005 closed 1 year ago

andersy005 commented 1 year ago

This PR adds a configurable option for maximum payload size supported by the server. When the client requests chunk sizes larger than the max payload size, the server returns a 400 error.

Cc @katamartin

❯ http -v 'http://127.0.0.1:8000/ncsa.osn.xsede.org/Pangeo%2Fpangeo-forge%2FWOA_1degree_monthly-feedstock%2Fwoa18-1deg-monthly.zarr%2FA_an/0.0.0.0' 'chunks: A_an=12,30,180,256, A_dd=12,30,180,256, A_gp=12,30,180,256, A_ma=12,30,180,256, A_mn=12,30,180,256, A_oa=12,30,180,256, A_sd=12,30,180,256, A_se=12,30,180,256, I_an=12,30,180,256, I_dd=12,30,180,256, I_gp=12,30,180,256, I_ma=12,30,180,256, I_mn=12,30,180,256, I_oa=12,30,180,256, I_sd=12,30,180,256, I_se=12,30,180,256, o_an=12,30,180,256, o_dd=12,30,180,256, o_gp=12,30,180,256, o_ma=12,30,180,256, o_mn=12,30,180,256, o_oa=12,30,180,256, o_sd=12,30,180,256, o_se=12,30,180,256, s_an=12,30,180,256, s_dd=12,30,180,256, s_gp=12,30,180,256, s_ma=12,30,180,256, s_mn=12,30,180,256, s_oa=12,30,180,256, s_sd=12,30,180,256, s_se=12,30,180,256, t_an=12,30,180,256, t_dd=12,30,180,256, t_gp=12,30,180,256, t_ma=12,30,180,256, t_mn=12,30,180,256, t_oa=12,30,180,256, t_sd=12,30,180,256, t_se=12,30,180,256'
GET /ncsa.osn.xsede.org/Pangeo%2Fpangeo-forge%2FWOA_1degree_monthly-feedstock%2Fwoa18-1deg-monthly.zarr%2FA_an/0.0.0.0 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: 127.0.0.1:8000
User-Agent: HTTPie/3.2.1
chunks: A_an=12,30,180,256, A_dd=12,30,180,256, A_gp=12,30,180,256, A_ma=12,30,180,256, A_mn=12,30,180,256, A_oa=12,30,180,256, A_sd=12,30,180,256, A_se=12,30,180,256, I_an=12,30,180,256, I_dd=12,30,180,256, I_gp=12,30,180,256, I_ma=12,30,180,256, I_mn=12,30,180,256, I_oa=12,30,180,256, I_sd=12,30,180,256, I_se=12,30,180,256, o_an=12,30,180,256, o_dd=12,30,180,256, o_gp=12,30,180,256, o_ma=12,30,180,256, o_mn=12,30,180,256, o_oa=12,30,180,256, o_sd=12,30,180,256, o_se=12,30,180,256, s_an=12,30,180,256, s_dd=12,30,180,256, s_gp=12,30,180,256, s_ma=12,30,180,256, s_mn=12,30,180,256, s_oa=12,30,180,256, s_sd=12,30,180,256, s_se=12,30,180,256, t_an=12,30,180,256, t_dd=12,30,180,256, t_gp=12,30,180,256, t_ma=12,30,180,256, t_mn=12,30,180,256, t_oa=12,30,180,256, t_sd=12,30,180,256, t_se=12,30,180,256

HTTP/1.1 400 Bad Request
content-length: 111
content-type: application/json
date: Fri, 10 Feb 2023 19:12:36 GMT
server: uvicorn

{
    "detail": "Chunk with 63.28 MiB and shape (12, 30, 180, 256) exceeds server's payload size limit of 5.72 MiB."
}