How to escape AMP query parameter right?

[danil-smirnov]

Hi,

I'm trying to query AMP instance withawscurl like this:

docker run --rm -it okigan/awscurl -i --access_key $AWS_ACCESS_KEY_ID --secret_key $AWS_SECRET_ACCESS_KEY --session_token $AWS_SESSION_TOKEN --region eu-central-1 --service aps $AMP_QUERY_ENDPOINT'?query=http_request_duration_seconds_bucket{}'

It works fine until I add a parameter between curly brackets:

docker run --rm -it okigan/awscurl -i --access_key $AWS_ACCESS_KEY_ID --secret_key $AWS_SECRET_ACCESS_KEY --session_token $AWS_SESSION_TOKEN --region eu-central-1 --service aps $AMP_QUERY_ENDPOINT'?query=http_request_duration_seconds_bucket{status="2xx"}'

I'm getting InvalidQueryStringException error in the latter case, though it works fine in the Grafana preview.

I tried different escape methods but haven't found a working one.

[okigan]

try to escape the parameters (you could use https://www.urlencoder.org/), so something like this:

docker run --rm -it okigan/awscurl -i --access_key "$AWS_ACCESS_KEY_ID" --secret_key "$AWS_SECRET_ACCESS_KEY" --session_token "$AWS_SESSION_TOKEN" --region eu-central-1 --service aps "${AMP_QUERY_ENDPOINT}?query=http_request_duration_seconds_bucket%7Bstatus%3D%222xx%22%7D"

On Fri, May 12, 2023 at 9:15 AM Danil Smirnov @.***> wrote:

Hi,

I'm trying to query AMP instance with awscurl like this:

docker run --rm -it okigan/awscurl -i --access_key $AWS_ACCESS_KEY_ID --secret_key $AWS_SECRET_ACCESS_KEY --session_token $AWS_SESSION_TOKEN --region eu-central-1 --service aps $AMP_QUERY_ENDPOINT'?query=http_request_duration_seconds_bucket{}'

It works fine until I add a parameter between curly brackets:

docker run --rm -it okigan/awscurl -i --access_key $AWS_ACCESS_KEY_ID --secret_key $AWS_SECRET_ACCESS_KEY --session_token $AWS_SESSION_TOKEN --region eu-central-1 --service aps $AMP_QUERY_ENDPOINT'?query=http_request_duration_seconds_bucket{status="2xx"}'

I'm getting InvalidQueryStringException error in the latter case, though it works fine in the Grafana preview.

I tried different escape methods but haven't found a working one.

— Reply to this email directly, view it on GitHub https://github.com/okigan/awscurl/issues/169, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADUYXSBTBPCN5NKMZZXLNDXFZAWXANCNFSM6AAAAAAX7S5SKI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[danil-smirnov]

@okigan If I do this, I'm getting InvalidSignatureException error:

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/workspaces/ws-f77bee0c-0494-4267-b64c-91c938eb734b/api/v1/query\nquery=http_request_duration_seconds_bucket%7Bstatus%3D%222xx%22%7D\nhost:aps-workspaces.eu-central-1.amazonaws.com\nx-amz-date:20230512T150529Z\nx-amz-security-token:IQoJb3JpZ2luX2VjEJ///////////wEaDGV1LWNlbnRyYWwtMSJIMEYCIQD7O5A8aNIqZzI2xLb7tXmg3EjnqFSOEjcL0g+z9+rN3AIhAN05+QRO710db4H7y1EL/e8xfYNAriEtVFhlcTcGqGBCKqEDCLj//////////wEQABoMNzM3NDE1OTk2OTg4Igx4CGmpfUX1O3O2jskq9QLNUIN90sCdL25chH49AuK+YQ1pzsIE6BkQJgMpILs20GHZLQHPU33feHyRYe4rip7KidTF4q8aC/YnUnIKGdqDVlRnFjV4b4cXrUwIcgV9afbAyHtCcb1dMFnhzzD98IUCanKNI1MXPxfYhoLBUk2977CZpA4h5KOz0Sm36v9iXWxHUUhvc2Snb5XMvN0HPGaIqN1arIpT0gLj9LrxnDLvN29+/rj+Si/TxOiAw9KFBUNAnvNyLvIjDs/BU77utrB4gQ54QhHC7c22Nxwyc5uTKCGTQl3a9MxT809HMzO8jZaDHydQyRTa5s/hXYxEH6anDt4oQ5f1voDw2+gaQGNEhURc2XA80TMXMEZrCHnZql2aENh3Lb6DYMFq1jI39s6zM/E0pzz+MAsYzwYg8HP7p8Qpkh0OEhueX4OqdkX4eKHlJOma/EZY5R+XJd1LDTwBSnNpNM70Wp2gd4SJQ23FgW9Ec/hgWyeXXGF43DRzNUngowolMKLJ96IGOqUBymR8DkPg5jacAj8dVOGu+fgjCFAaN7hWWZR0A4RjbBPwXbRQV9R2jt6eLJe03sWTHseWUTkzDoxGjqhMseuX/BlgshQwguACzDwHFk62Jns0RpsQa4zKW4SrLaGxygDpFhodZkzAg6Wvo3IHYAPVVc8QObPB1CeVgNU0XdVvPGbjHl9rN5bhv5CX7kn8i1TTDOEaQKxobWHG837EKS8yzej9s6Io\n\nhost;x-amz-date;x-amz-security-token\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20230512T150529Z\n20230512/eu-central-1/aps/aws4_request\n119ac3cb15e44925ac54ecb3a66e028afe8069f1a59810eb65b4651a36d668ec'\n"}

[okigan]

oh, interesting. I'd need to debug it.

mean time try escaping/encoding the '=' character

On Fri, May 12, 2023 at 10:08 AM Danil Smirnov @.***> wrote:

@okigan https://github.com/okigan If I do this, I'm getting InvalidSignatureException error:

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/workspaces/ws-f77bee0c-0494-4267-b64c-91c938eb734b/api/v1/query\nquery=http_request_duration_seconds_bucket%7Bstatus%3D%222xx%22%7D\nhost: aps-workspaces.eu-central-1.amazonaws.com\nx-amz-date:20230512T150529Z\nx-amz-security-token:IQoJb3JpZ2luX2VjEJ///////////wEaDGV1LWNlbnRyYWwtMSJIMEYCIQD7O5A8aNIqZzI2xLb7tXmg3EjnqFSOEjcL0g+z9+rN3AIhAN05+QRO710db4H7y1EL/e8xfYNAriEtVFhlcTcGqGBCKqEDCLj//////////wEQABoMNzM3NDE1OTk2OTg4Igx4CGmpfUX1O3O2jskq9QLNUIN90sCdL25chH49AuK+YQ1pzsIE6BkQJgMpILs20GHZLQHPU33feHyRYe4rip7KidTF4q8aC/YnUnIKGdqDVlRnFjV4b4cXrUwIcgV9afbAyHtCcb1dMFnhzzD98IUCanKNI1MXPxfYhoLBUk2977CZpA4h5KOz0Sm36v9iXWxHUUhvc2Snb5XMvN0HPGaIqN1arIpT0gLj9LrxnDLvN29+/rj+Si/TxOiAw9KFBUNAnvNyLvIjDs/BU77utrB4gQ54QhHC7c22Nxwyc5uTKCGTQl3a9MxT809HMzO8jZaDHydQyRTa5s/hXYxEH6anDt4oQ5f1voDw2+gaQGNEhURc2XA80TMXMEZrCHnZql2aENh3Lb6DYMFq1jI39s6zM/E0pzz+MAsYzwYg8HP7p8Qpkh0OEhueX4OqdkX4eKHlJOma/EZY5R+XJd1LDTwBSnNpNM70Wp2gd4SJQ23FgW9Ec/hgWyeXXGF43DRzNUngowolMKLJ96IGOqUBymR8DkPg5jacAj8dVOGu+fgjCFAaN7hWWZR0A4RjbBPwXbRQV9R2jt6eLJe03sWTHseWUTkzDoxGjqhMseuX/BlgshQwguACzDwHFk62Jns0RpsQa4zKW4SrLaGxygDpFhodZkzAg6Wvo3IHYAPVVc8QObPB1CeVgNU0XdVvPGbjHl9rN5bhv5CX7kn8i1TTDOEaQKxobWHG837EKS8yzej9s6Io\n\nhost;x-amz-date;x-amz-security-token\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20230512T150529Z\n20230512/eu-central-1/aps/aws4_request\n119ac3cb15e44925ac54ecb3a66e028afe8069f1a59810eb65b4651a36d668ec'\n"}

— Reply to this email directly, view it on GitHub https://github.com/okigan/awscurl/issues/169#issuecomment-1545894134, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADUYXXL2OJTRM2D2HXJXHTXFZG5ZANCNFSM6AAAAAAX7S5SKI . You are receiving this because you were mentioned.Message ID: @.***>

[danil-smirnov]

The error is the same if I do

docker run --rm -it okigan/awscurl -i --access_key "$AWS_ACCESS_KEY_ID" --secret_key "$AWS_SECRET_ACCESS_KEY" --session_token "$AWS_SESSION_TOKEN" --region eu-central-1 --service aps "${AMP_QUERY_ENDPOINT}?query%3Dhttp_request_duration_seconds_bucket%7Bstatus%3D%222xx%22%7D"

I saw some closed issues mentioning this, I suppose we have a bug here

[okigan]

Please add link(s) to relevant ones here - thanks!

[danil-smirnov]

@okigan Here it is:

https://github.com/okigan/awscurl/issues?q=is%3Aissue+escape

[okigan]

@danil-smirnov This is a tricky issue, especially for existing users...and number existing unit tests

Could you confirm source in this PR resolve your issue (check out the PR branch) and test similarly as shown in the top comment, but with AMP query: https://github.com/okigan/awscurl/pull/171#issue-1708219549

[danil-smirnov]

@okigan I've built a Docker image from branch issue/169 and tried all the variants but I still got InvalidQueryStringException or InvalidSignatureException errors

[alediaferia]

Is there any update for this or any recommended alternative approach?

[okigan]

Yeah with the PR above we (@danil-smirnov and I) were able to get it to work.

@alediaferia are you using branch build? with/without docker?

[yuwuc]

@okigan I think the issue is in this line. Double url encoding. After removing function aws_url_encode(), my prometheus query works.

Sample command:

awscurl -X POST --region us-west-2 --service aps "${AMP_QUERY_ENDPOINT}query?query=group%20by%28__name__%29%20%28%7B__name__%21%3D%22%22%7D%29"

You could see the query string changed after enabling debugging: ('\n' 'CANONICAL REQUEST = POST\n' '/workspaces/-----/api/v1/query\n' 'query=group%2520by%2528__name__%2529%2520%2528%257B__name__%2521%253D%2522%2522%257D%2529\n' 'host:aps-workspaces.us-west-2.amazonaws.com\n' 'x-amz-date:20230803T200239Z\n'

The query string changed from group%20by%28__name__%29%20%28%7B__name__%21%3D%22%22%7D%29 to group%2520by%2528__name__%2529%2520%2528%257B__name__%2521%253D%2522%2522%257D%2529

[okigan]

Yes the issue is about encoding, question is who suppose to do it. In your example you already encoded them them before calling the awscurl but some users don’t. I’ve reread this article [1] by maker of curl, but did not arrive at succinct conclusion. Give it a shot, tell me what make sense (with reference). [1] https://daniel.haxx.se/blog/2016/05/11/my-url-isnt-your-url/On Aug 3, 2023, at 3:41 PM, Yu-Wu Chu @.***> wrote: @okigan I think the issue is in this line. Double url encode. After removing function aws_url_encode(), my prometheus query works. Sample command: awscurl -X POST --region us-west-2 --service aps "${AMP_QUERY_ENDPOINT}query?query=group%20by%28name%29%20%28%7Bname%21%3D%22%22%7D%29" You could see the query string changed after enabling debugging: ('\n' 'CANONICAL REQUEST = POST\n' '/workspaces/-----/api/v1/query\n' 'query=group%2520by%2528name%2529%2520%2528%257Bname%2521%253D%2522%2522%257D%2529\n' 'host:aps-workspaces.us-west-2.amazonaws.com\n' 'x-amz-date:20230803T200239Z\n' The query string changed from group%20by%28name%29%20%28%7Bname%21%3D%22%22%7D%29 to group%2520by%2528name%2529%2520%2528%257Bname%2521%253D%2522%2522%257D%2529

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[yuwuc]

@okigan Thanks for sharing the article. It is interesting. You are right. This might need to think more about it.

[yuwuc]

I found another work around. Hopefully, this helps who faces this issue via POST and move query string to body.

awscurl -X POST --region us-west-2 --service aps "${AMP_QUERY_ENDPOINT}/query" -d 'query=group by(__name__) ({__name__!=""})' --header 'Content-Type: application/x-www-form-urlencoded'

[okigan]

I think that's a great find! -- side steps this whole issue

On Fri, Aug 4, 2023 at 12:58 PM Yu-Wu Chu @.***> wrote:

I found another work around. Hopefully, this helps who faces this issue via POST and move query string to body.

awscurl -X POST --region us-west-2 --service aps "${AMP_QUERY_ENDPOINT}/query" -d 'query=group by(name) ({name!=""})' --header 'Content-Type: application/x-www-form-urlencoded'

— Reply to this email directly, view it on GitHub https://github.com/okigan/awscurl/issues/169#issuecomment-1665984897, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADUYXWANFXUJGWF5G6FRGLXTUZ2VANCNFSM6AAAAAAX7S5SKI . You are receiving this because you were mentioned.Message ID: @.***>