Latest docker release broken

[gauteh]

Hi,

I recently pulled in hyrax:latest from docker and it fails to apply constraints to a MEPS dataset from met.no. Downgrading to 1.16.1 fixes the issue. Fails when doing ncdump on DAP/2.

Here is a log from various attempts using curl:

dars/dars on  master [$] via 🦀 v1.46.0-nightly took 2s
❯ curl http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x%5fwind%5fml.x%5fwind%5fml%5c1:66%5d%5c0:64%5d%5c0%5d%5c0:28386%5d
Error {
    code = 400;
    message = "libdap error transmitting DataDDS: Constraint expression parse error: the expression referenced a identifier not found in the dataset.";
}
dars/dars on  master [$] via 🦀 v1.46.0-nightly
❯ curl http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5c1:66%5d%5c0:64%5d%5c0%5d%5c0:28386%5d
Error {
    code = 400;
    message = "libdap error transmitting DataDDS: Constraint expression parse error: the expression referenced a identifier not found in the dataset.";
}
dars/dars on  master [$] via 🦀 v1.46.0-nightly
❯ curl http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml
Dataset {
    Structure {
        Int32 x_wind_ml[time = 67][hybrid = 65][y = 1][x = 28387];
    } x_wind_ml;
} meps_det_vc_2_5km_latest.nc;
Data:
^C
dars/dars on  master [$] via 🦀 v1.46.0-nightly
❯ curl http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5C1:66%5D       Error {
    code = 400;
    message = "libdap error transmitting DataDDS: Constraint expression parse error: the expression referenced a identifier not found in the dataset.";
}

fails using ncdump and works again with ncdump on 1.16.1.

Cheers, Gaute

[ndp-opendap]

Hi Gaute,

It looks to me like like there is a problem with the way that you are encoding your URLs.

When I take the provided example URL:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x%5fwind%5fml.x%5fwind%5fml%5c1:66%5d%5c0:64%5d%5c0%5d%5c0:28386%5d

And put it through a URL decoder: https://www.urldecoder.io

I get back this:

  http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]\0:64]\0]\0:28386]

Where we can see that all occurrences of "[" have been encoded as %5c (which is "\") rather than %5b.

I would have expected it to be:

  http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml[1:66][0:64][0][0:28386]

Does this (possibly corrected) URL work for you?

  http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5B1%3A66%5D%5B0%3A64%5D%5B0%5D%5B0%3A28386%5D

Similarly:

  http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5c1:66%5d%5c0:64%5d%5c0%5d%5c0:28386%5d

Decodes to:

  http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]\0:64]\0]\0:28386]

And:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5C1:66%5D

Decodes to:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]

Also, this interchange:

❯ curl http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml

Dataset {
    Structure {
        Int32 x_wind_ml[time = 67][hybrid = 65][y = 1][x = 28387];
    } x_wind_ml;
} meps_det_vc_2_5km_latest.nc;
Data:
^C

Appears to have been in process when it was terminated with a Ctrl-C.

Please let me know if that sorts anything out for you, or if you are still having problems.

Sincerely,

Nathan

[gauteh]

Yes. I have always used unencoded constraint expressions. Has this changed? Note that ncdump also fails, ncdump encodes correctly (as far as I know). The curl examples I provided might be flawed, I can try to do some proper tests with curl as well. I have done unencoded curl requests untill latest without problems.

man. 6. jul. 2020, 17:09 skrev Nathan Potter notifications@github.com:

Hi Gaute,

It looks to me like like there is a problem with the way that you are encoding your URLs.

When I take the provided example URL:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x%5fwind%5fml.x%5fwind%5fml%5c1:66%5d%5c0:64%5d%5c0%5d%5c0:28386%5d

And put it through a URL decoder: https://www.urldecoder.io

I get back this:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]\0:64]\0]\0:28386]

Where we can see that all occurrences of "[" have been encoded as %5c (which is "\") rather than %5b.

I would have expected it to be:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml[1:66][0:64][0][0:28386]

Does this (possibly corrected) URL work for you?

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5B1%3A66%5D%5B0%3A64%5D%5B0%5D%5B0%3A28386%5D

Similarly:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5c1:66%5d%5c0:64%5d%5c0%5d%5c0:28386%5d

Decodes to:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]\0:64]\0]\0:28386]

And:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5C1:66%5D

Decodes to:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]

Also, this interchange:

❯ curl http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml

Dataset {

Structure {

    Int32 x_wind_ml[time = 67][hybrid = 65][y = 1][x = 28387];

} x_wind_ml;

} meps_det_vc_2_5km_latest.nc;

Data:

^C

Appears to have been in process when it was terminated with a Ctrl-C.

Please let me know if that sorts anything out for you, or if you are still having problems.

Sincerely,

Nathan

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OPENDAP/hyrax-docker/issues/29#issuecomment-654296514, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN36423ZHBUXT4OJND5ILR2HSKXANCNFSM4ORN2ZGQ .

[gauteh]

It only fails when constraining. Not unconstrained, so that's why the last example works.

man. 6. jul. 2020, 17:20 skrev Gaute Hope eg@gaute.vetsj.com:

Yes. I have always used unencoded constraint expressions. Has this changed? Note that ncdump also fails, ncdump encodes correctly (as far as I know). The curl examples I provided might be flawed, I can try to do some proper tests with curl as well. I have done unencoded curl requests untill latest without problems.

man. 6. jul. 2020, 17:09 skrev Nathan Potter notifications@github.com:

Hi Gaute,

It looks to me like like there is a problem with the way that you are encoding your URLs.

When I take the provided example URL:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x%5fwind%5fml.x%5fwind%5fml%5c1:66%5d%5c0:64%5d%5c0%5d%5c0:28386%5d

And put it through a URL decoder: https://www.urldecoder.io

I get back this:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]\0:64]\0]\0:28386]

Where we can see that all occurrences of "[" have been encoded as %5c (which is "\") rather than %5b.

I would have expected it to be:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml[1:66][0:64][0][0:28386]

Does this (possibly corrected) URL work for you?

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5B1%3A66%5D%5B0%3A64%5D%5B0%5D%5B0%3A28386%5D

Similarly:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5c1:66%5d%5c0:64%5d%5c0%5d%5c0:28386%5d

Decodes to:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]\0:64]\0]\0:28386]

And:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml%5C1:66%5D

Decodes to:

http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml\1:66]

Also, this interchange:

❯ curl http://localhost:8003/opendap/meps_det_vc_2_5km_latest.nc.dods?x_wind_ml.x_wind_ml

Dataset {

Structure {

    Int32 x_wind_ml[time = 67][hybrid = 65][y = 1][x = 28387];

} x_wind_ml;

} meps_det_vc_2_5km_latest.nc;

Data:

^C

Appears to have been in process when it was terminated with a Ctrl-C.

Please let me know if that sorts anything out for you, or if you are still having problems.

Sincerely,

Nathan

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OPENDAP/hyrax-docker/issues/29#issuecomment-654296514, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN36423ZHBUXT4OJND5ILR2HSKXANCNFSM4ORN2ZGQ .

[ndp-opendap]

Uncoded URLs that contain characters that should be encoded (like "[" and "]") have been rejected by Tomcat since roughly June 2018 (Tomcat-8.0.53). Certainly the version of Tomcat in the "latest" docker container will reject them, returning only an HTTP status of 400 and no response body.

Yes. I have always used unencoded constraint expressions. Has this changed? Note that ncdump also fails, ncdump encodes correctly (as far as I know).

I think that this very much depends on what version of of ncdump (the netcdf-c lib) that you are using, I believe they only added this in the last year, about the same time that Tomcat cracked down on unencoded URL content.

The curl examples I provided might be flawed, I can try to do some proper tests with curl as well. I have done unencoded curl requests until latest without problems.

[gauteh]

Hi. You're right. Correctly encoding the URL does go through on hyrax:latest. ncdump v4.7.4 works, while 4.6.0 does not. Using hyrax:1.16.1 it works without encoding the brackets. The underscores do not seem to be required to be encoded.

On Mon, Jul 6, 2020 at 5:30 PM Nathan Potter notifications@github.com wrote:

Uncoded URLs that contain characters that should be encoded (like "[" and "]") have been rejected by Tomcat since roughly June 2018 (_Tomcat-8.0.5_3). Certainly the version of Tomcat in the "latest" docker container will reject them, returning only an HTTP status of 400 and no response body.

Yes. I have always used unencoded constraint expressions. Has this changed? Note that ncdump also fails, ncdump encodes correctly (as far as I know).

I think that this very much depends on what version of of ncdump (the netcdf-c lib) that you are using, I believe they only added this in the last year, about the same time that Tomcat cracked down on unencoded URL content.

The curl examples I provided might be flawed, I can try to do some proper tests with curl as well. I have done unencoded curl requests until latest without problems.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OPENDAP/hyrax-docker/issues/29#issuecomment-654308555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN364R3IM7DAPG56GPGEDR2HU2LANCNFSM4ORN2ZGQ .

[ndp-opendap]

Hi Gaute,

The underscores are considered "safe" characters, while the square brackets are considered unsafe. This page: https://perishablepress.com/stop-using-unsafe-characters-in-urls has a nice character encoding chart that summarizes some of the concerns.

Also - just to be clear, the tag latest in our docker hub is always the most recent numbered release (1.16.1 at the moment) we also have a snapshot tag which is the most recent CI build of master branch with passing tests.

Sincerely,

Nathan

[gauteh]

It seems like it works like it should at any rate. But I'm pretty sure there is a difference between 1.16.1 and latest. Maybe latest points to 1.16.2?

On Mon, Jul 6, 2020 at 8:45 PM Nathan Potter notifications@github.com wrote:

Hi Gaute,

The underscores are considered "safe" characters, while the square brackets are considered unsafe. This page: https://perishablepress.com/stop-using-unsafe-characters-in-urls has a nice character encoding chart that summarizes some of the concerns.

Also - just to be clear, the tag latest in our docker hub is always the most recent numbered release (1.16.1 at the moment) we also have a snapshot tag which is the most recent CI build of master branch with passing tests.

Sincerely,

Nathan

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OPENDAP/hyrax-docker/issues/29#issuecomment-654404278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN363C3ACKOAA64A3MNSDR2ILVLANCNFSM4ORN2ZGQ .

[ndp-opendap]

My mistake. Hyrax is at 1.16.2 now.

[gauteh]

:+1: I consider the issue resolved on my part.

On Mon, Jul 6, 2020 at 8:49 PM Nathan Potter notifications@github.com wrote:

My mistake. Hyrax is at 1.16.2 now.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/OPENDAP/hyrax-docker/issues/29#issuecomment-654405971, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN36YGDPNFTPJQQBEJFDTR2IMDZANCNFSM4ORN2ZGQ .

[gauteh]

Thanks for you help.

[ndp-opendap]

Anytime! I'm glad it was a simple thing.