search

okigan / awscurl

curl-like access to AWS resources with AWS Signature Version 4 request signing.
MIT License
735 stars 91 forks source link

[FEATURE] enhance awscurl support create/send multipart POST REST request (from file) #194

Closed ghost closed 3 months ago

ghost commented 3 months ago

Versions

What's my problem ? First of all, this closed issue : https://github.com/opensearch-project/opensearch-cli/issues/76 Seems to be close to my problem, but I'm not using opensearch-cli curl. Only awscurl

I try to migrate dashboards objects from one opensearch cluster to another one. To do that I'm using api provided by opensearch : /_dashboards/api/saved_objects/_import?overwrite=true But I get errors when I'm trying to post my objects.

How that's done ? I'm exporting the objects via the opensearch UI (From my first cluster). Dashboards Management > Saved objects > Export (when my dashboard is selected). And I get my objects in a .ndjson file. After that i'm posting my .ndjson file with awscurl :

Without Content Type header

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'
{
   "statusCode":415,
   "error":"Unsupported Media Type",
   "message":"Unsupported Media Type"
}

Without Boundary

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'Content-Type: multipart/form-data' \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'
{
   "statusCode":400,
   "error":"Bad Request",
   "message":"Invalid content-type header: multipart missing boundary"
}

With a random boundary

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'Content-Type: multipart/form-data; boundary=---------BOUNDARY;' \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'

image

image

By only using curl, and don't put Content Type header, it works fine.

curl -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H "osd-xsrf: true"
  -F "file=@data/test_dashboard.ndjson"
{
   "successCount":6,
   "success":true,
   "successResults":[
      {
         "type":"index-pattern",
         "id":"610267e0-7f21-11ee-80e6-370892a64263",
         "meta":{
            "title":"documents-metadata-v2",
            "icon":"indexPatternApp"
         },
         "overwrite":true
      },
      {
         "type":"visualization",
         "id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b",
         "meta":{
            "title":"count by cat_doc",
            "icon":"visualizeApp"
         },
         "overwrite":true
      },
      {
         "type":"index-pattern",
         "id":"d3d7af60-4c81-11e8-b3d7-01146121b73d",
         "meta":{
            "title":"opensearch_dashboards_sample_data_flights",
            "icon":"indexPatternApp"
         },
         "overwrite":true
      },
      {
         "type":"search",
         "id":"571aaf70-4c88-11e8-b3d7-01146121b73d",
         "meta":{
            "title":"[Flights] Flight Log",
            "icon":"discoverApp"
         },
         "overwrite":true
      },
      {
         "type":"visualization",
         "id":"94003870-dbb5-11ee-81ba-b7ae96977aa8",
         "meta":{
            "title":"test_object_dashboard",
            "icon":"visualizeApp"
         },
         "overwrite":true
      },
      {
         "type":"dashboard",
         "id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b",
         "meta":{
            "title":"test_dashboard",
            "icon":"dashboardApp"
         },
         "overwrite":true
      }
   ]
}
okigan commented 3 months ago

Seems like last curl command works without multipart ?  Let’s have a look at what it decides to send: please rerun it with -v parameter and post more details. On Mar 8, 2024, at 5:26 AM, llasherme @.***> wrote: Versions

OS :

cURL => via Windows 10 awscurl => via gitlab runner (Docker)

OpenSearch 2.11

What's my problem ? First of all, this closed issue : opensearch-project/opensearch-cli#76 Seems to be close to my problem, but I'm not using opensearch-cli curl. Only awscurl I try to migrate dashboards objects from one opensearch cluster to another one. To do that I'm using api provided by opensearch : /_dashboards/api/saved_objects/_import?overwrite=true But I get errors when I'm trying to post my objects. How that's done ? I'm exporting the objects via the opensearch UI (From my first cluster). Dashboards Management > Saved objects > Export (when my dashboard is selected). And I get my objects in a .ndjson file. After that i'm posting my .ndjson file with awscurl : Without Content Type header awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \ -H 'osd-xsrf: true' \ -d '${dashboard_content}' { "statusCode":415, "error":"Unsupported Media Type", "message":"Unsupported Media Type" } Without Boundary awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \ -H 'Content-Type: multipart/form-data' \ -H 'osd-xsrf: true' \ -d '${dashboard_content}' { "statusCode":400, "error":"Bad Request", "message":"Invalid content-type header: multipart missing boundary" } With a random boundary awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \ -H 'Content-Type: multipart/form-data; boundary=---------BOUNDARY;' \ -H 'osd-xsrf: true' \ -d '${dashboard_content}'

image.png (view on web) image.png (view on web) By only using curl, and don't put Content Type header, it works fine. curl -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \ -H "osd-xsrf: true" -F @.***/test_dashboard.ndjson" { "successCount":6, "success":true, "successResults":[ { "type":"index-pattern", "id":"610267e0-7f21-11ee-80e6-370892a64263", "meta":{ "title":"documents-metadata-v2", "icon":"indexPatternApp" }, "overwrite":true }, { "type":"visualization", "id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b", "meta":{ "title":"count by cat_doc", "icon":"visualizeApp" }, "overwrite":true }, { "type":"index-pattern", "id":"d3d7af60-4c81-11e8-b3d7-01146121b73d", "meta":{ "title":"opensearch_dashboards_sample_data_flights", "icon":"indexPatternApp" }, "overwrite":true }, { "type":"search", "id":"571aaf70-4c88-11e8-b3d7-01146121b73d", "meta":{ "title":"[Flights] Flight Log", "icon":"discoverApp" }, "overwrite":true }, { "type":"visualization", "id":"94003870-dbb5-11ee-81ba-b7ae96977aa8", "meta":{ "title":"test_object_dashboard", "icon":"visualizeApp" }, "overwrite":true }, { "type":"dashboard", "id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b", "meta":{ "title":"test_dashboard", "icon":"dashboardApp" }, "overwrite":true } ] }

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>

ghost commented 3 months ago

Yes, cURL doesn't need to explicit the Content Type. Here's the cURL with verbose option :

curl -v -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
-H "osd-xsrf: true" \
-F "file=@data/test_dashboard.ndjson"

And the result :


Note: Unnecessary use of -X or --request, POST is already inferred.
* Uses proxy env variable no_proxy == '****'
*   Trying [::1]:8080...
*   Trying ${domain_ip}:8080...
* Connected to localhost (127.0.0.1) port 8080
> POST /_dashboards/api/saved_objects/_import?overwrite=true HTTP/1.1
> Host: ${domain_url}
> User-Agent: curl/8.4.0
> Accept: */*
> osd-xsrf: true
> Content-Length: 40384
> Content-Type: multipart/form-data; boundary=------------------------wOWPtrGc1TrXmDPpX5GfNt
>
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Length: 947
< Content-Type: application/json; charset=utf-8
< Date: Tue, 12 Mar 2024 09:44:45 GMT
< Osd-Name: osd
< Set-Cookie: security_authentication_saml1=Fe26.2**838c01d554bcb2b7d4e5ead20dafc498fcdd74e1faf6b74c5d1348a2326b713c*D_5paBXbChAFcjTpZ00nLg*YUmGokrp_eV_EEv_2S0NZecOyi1Lwhrgas5VLH69YDC6nBJDau20hsaz8ipoTcUPxMS2AxQoGXqv5bUMo8sY8ulz783TWCck7iibWpBdYn_FvqXpBoCu9C23MAD-v6CDe7CJ5ch2KczWBZhnI-jc_z53Lz-Dt62h2PQ1OAAq7-He5QR7Jx-lg87XpaWpCL2A0-uAvfASBUoZ7ui2IvPHuLmw5myZ5acmFZsxpgg9QLrRKlBlUvpNRz0-rG6EZY2uI1UoVRnsaYFLUkRKe3YOZYSCsvLWxGDi6XBEUfdLH9bl3WSV1u0eDdAEvt0qViIcp8Njt0_iT1iyQiHR6ju-Xg**8def9b476d52a739f02c24f8ed8783b8585d01177dcd04e624d2ec17b0a17c61*nVfuzR87kXvFQPaesZZZfY1aPWiT87yLITvEMUJOj5s; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication_saml2=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication_saml3=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication=Fe26.2**59193956774e2ff6b56087734d72a99abd64c27811b759e7c10e85f995ae4792*CFtdPl74zukFptAGTWbAXw*bHObA_QMNUiaQI8V471pQhS_Z2GRkmQuy4-G6e5ycbyBSGZ_8W6ThTG7f3SlTgtQLD5AmqSa6mGKeZ5j6JJZoY0zTX2T5Zimkr8JWEljMBIlFOK_G-SSNmEtZPVuKHUfbKOVH-4_6WlW6DVVQO7rGYe9dnm615ssNvTuzFQNJ8BTZ5HVXutfXhMrOzipcs1nh4A2K6PaZnbT-RNpS823WT_JVmdfNcrHXj2xWfjYbHQLC2gv5FTcOHAMd0yPCDS0**9e47d862c46786b7011a75ddbde639601e554e9a30ca09366ede38184d642392*vsovYjoNWeD0yiX3QUmj3ryV1V0IJTuejNc3rBMOLP8; Secure; HttpOnly; Path=/_dashboards
< Strict-Transport-Security: max-age=31536000
< X-Amzn-Requestid: a56e2412-d6f8-4119-84ac-9eead1de5731
<
{"successCount":6,"success":true,"successResults":[{"type":"index-pattern","id":"610267e0-7f21-11ee-80e6-370892a64263","meta":{"title":"documents-metadata-v2","icon":"indexPatternApp"},"overwrite":true},{"type":"visualization","id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b","meta":{"title":"count by cat_doc","icon":"visualizeApp"},"overwrite":true},{"type":"index-pattern","id":"d3d7af60-4c81-11e8-b3d7-01146121b73d","meta":{"title":"opensearch_dashboards_sample_data_flights","icon":"indexPatternApp"},"overwrite":true},{"type":"search","id":"571aaf70-4c88-11e8-b3d7-01146121b73d","meta":{"title":"[Flights] Flight Log","icon":"discoverApp"},"overwrite":true},{"type":"visualization","id":"94003870-dbb5-11ee-81ba-b7ae96977aa8","meta":{"title":"test_object_dashboard","icon":"visualizeApp"},"overwrite":true},{"type":"dashboard","id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b","meta":{"title":"test_dashboard","icon":"dashboardApp"},"overwrite":true}]}

* Connection #0 to host ${domain}left intact
```^

As we can see, cURL putting itself the Content Type to multipart/form-data and setting the boundary to my data.
okigan commented 3 months ago

So, I think -F has extra processing to send a request with "data" and "files". the second part creates the multi-part sections.

So we'd need to create that option in awscurl

On Tue, Mar 12, 2024 at 4:53 AM llasherme @.***> wrote:

Yes, cURL doesn't need to explicit the Content Type. Here's the cURL with verbose option :

curl -v -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \ -H "osd-xsrf: true" \ -F @.***/test_dashboard.ndjson"

And the result :

Note: Unnecessary use of -X or --request, POST is already inferred.

  • Uses proxy env variable no_proxy == '****'

  • Trying [::1]:8080...

  • Trying ${domain_ip}:8080...

  • Connected to localhost (127.0.0.1) port 8080

    POST /_dashboards/api/saved_objects/_import?overwrite=true HTTP/1.1 Host: ${domain_url} User-Agent: curl/8.4.0 Accept: / osd-xsrf: true Content-Length: 40384 Content-Type: multipart/form-data; boundary=------------------------wOWPtrGc1TrXmDPpX5GfNt

  • We are completely uploaded and fine < HTTP/1.1 200 OK < Cache-Control: private, no-cache, no-store, must-revalidate < Content-Length: 947 < Content-Type: application/json; charset=utf-8 < Date: Tue, 12 Mar 2024 09:44:45 GMT < Osd-Name: osd < Set-Cookie: security_authentication_saml1=Fe26.2838c01d554bcb2b7d4e5ead20dafc498fcdd74e1faf6b74c5d1348a2326b713cD_5paBXbChAFcjTpZ00nLgYUmGokrp_eV_EEv_2S0NZecOyi1Lwhrgas5VLH69YDC6nBJDau20hsaz8ipoTcUPxMS2AxQoGXqv5bUMo8sY8ulz783TWCck7iibWpBdYn_FvqXpBoCu9C23MAD-v6CDe7CJ5ch2KczWBZhnI-jc_z53Lz-Dt62h2PQ1OAAq7-He5QR7Jx-lg87XpaWpCL2A0-uAvfASBUoZ7ui2IvPHuLmw5myZ5acmFZsxpgg9QLrRKlBlUvpNRz0-rG6EZY2uI1UoVRnsaYFLUkRKe3YOZYSCsvLWxGDi6XBEUfdLH9bl3WSV1u0eDdAEvt0qViIcp8Njt0_iT1iyQiHR6ju-Xg8def9b476d52a739f02c24f8ed8783b8585d01177dcd04e624d2ec17b0a17c61*nVfuzR87kXvFQPaesZZZfY1aPWiT87yLITvEMUJOj5s; Secure; HttpOnly; Path=/_dashboards < Set-Cookie: security_authentication_saml2=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards < Set-Cookie: security_authentication_saml3=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards < Set-Cookie: security_authentication=Fe26.259193956774e2ff6b56087734d72a99abd64c27811b759e7c10e85f995ae4792CFtdPl74zukFptAGTWbAXwbHObA_QMNUiaQI8V471pQhS_Z2GRkmQuy4-G6e5ycbyBSGZ_8W6ThTG7f3SlTgtQLD5AmqSa6mGKeZ5j6JJZoY0zTX2T5Zimkr8JWEljMBIlFOK_G-SSNmEtZPVuKHUfbKOVH-4_6WlW6DVVQO7rGYe9dnm615ssNvTuzFQNJ8BTZ5HVXutfXhMrOzipcs1nh4A2K6PaZnbT-RNpS823WT_JVmdfNcrHXj2xWfjYbHQLC2gv5FTcOHAMd0yPCDS09e47d862c46786b7011a75ddbde639601e554e9a30ca09366ede38184d642392*vsovYjoNWeD0yiX3QUmj3ryV1V0IJTuejNc3rBMOLP8; Secure; HttpOnly; Path=/_dashboards < Strict-Transport-Security: max-age=31536000 < X-Amzn-Requestid: a56e2412-d6f8-4119-84ac-9eead1de5731 < {"successCount":6,"success":true,"successResults":[{"type":"index-pattern","id":"610267e0-7f21-11ee-80e6-370892a64263","meta":{"title":"documents-metadata-v2","icon":"indexPatternApp"},"overwrite":true},{"type":"visualization","id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b","meta":{"title":"count by cat_doc","icon":"visualizeApp"},"overwrite":true},{"type":"index-pattern","id":"d3d7af60-4c81-11e8-b3d7-01146121b73d","meta":{"title":"opensearch_dashboards_sample_data_flights","icon":"indexPatternApp"},"overwrite":true},{"type":"search","id":"571aaf70-4c88-11e8-b3d7-01146121b73d","meta":{"title":"[Flights] Flight Log","icon":"discoverApp"},"overwrite":true},{"type":"visualization","id":"94003870-dbb5-11ee-81ba-b7ae96977aa8","meta":{"title":"test_object_dashboard","icon":"visualizeApp"},"overwrite":true},{"type":"dashboard","id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b","meta":{"title":"test_dashboard","icon":"dashboardApp"},"overwrite":true}]}

  • Connection #0 to host ${domain}left intact

As we can see, cURL putting itself the Content Type to multipart/form-data and setting the boundary to my data.

— Reply to this email directly, view it on GitHub https://github.com/okigan/awscurl/issues/194#issuecomment-1991218427, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADUYXTNJPP66DUNWI5GRJDYX3GBXAVCNFSM6AAAAABEMU6HUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGIYTQNBSG4 . You are receiving this because you commented.Message ID: @.***>

okigan commented 3 months ago

@llasherme try out this branch, his are the relevant steps:

git checkout feature/194
make venv
source venv/bin/activate
./run.sh -v -X POST -F name=@README.md https://httpbin.org/post

once that works try with your ES cluster

okigan commented 3 months ago

@llasherme interesting, try latest please (de473d83)

okigan commented 3 months ago

@llasherme - would love some context, did you run into any issues or something else came up?On Mar 18, 2024, at 3:44 AM, llasherme @.***> wrote: Closed #194 as not planned.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were assigned.Message ID: @.***>