search

abdolence / gcloud-sdk-rs

Async Google Cloud Platform (GCP) gRPC/REST APIs client implementation based on Tonic middleware and Reqwest.
Apache License 2.0
73 stars 22 forks source link

REST API: POST requests to Compute API fail with 411 (Length Required) #121

Open TmLev opened 10 months ago

TmLev commented 10 months ago

I'm trying to start a stopped/terminated compute instance:

let compute_config = google_rest_client.create_google_compute_v1_config().await?;
let request = gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodStartParams {
    project: "<PROJECT>".into(),
    instance: "<INSTANCE>".into(),
    zone: "<ZONE>".into(),
    ..Default::default()
};
let response =
    gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_start(
        &compute_config,
        request,
    )
    .await?;

Here's the response I get:

Error: ResponseError(ResponseContent { status: 411, content: "<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n  <title>Error 411 (Length Required)!!1</title>\n  <style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n  </style>\n  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>\n  <p><b>411.</b> <ins>That’s an error.</ins>\n  <p>POST requests require a <code>Content-length</code> header.  <ins>That’s all we know.</ins>\n", entity: None })

Stripped down:

status: 411
content: 411. That’s an error. POST requests require a Content-length header. That’s all we know.
abdolence commented 10 months ago

Hey,

I have almost the same working example here: https://github.com/abdolence/gcloud-sdk-rs/blob/d0320aea705c6df846918458b213a0739ff3ed4a/examples/gcs-rest-client/src/main.rs#L70

and it works without issues. In fact, I just copied your code and filled in my parameters and it seems to be working also as I expected.

So, something different in your environment and looking at your response it a bit weird, it returns HTML instead of more REST-friendly responses.

So, there are possible issues:

abdolence commented 10 months ago

Most likely this is related to the authentication so you have something off in your generated token.

TmLev commented 10 months ago

Hey! Thanks for your response :)

  • Please check if you authenticated correctly

I'm using service account saved as JSON. This service account should have the necessary permissions, but I'll double check.

  • Verify if you had enabled appropriate features
  • Using the latest version

Here's my Cargo.toml:

gcloud-sdk = { version = "0.24.2", features = ["google-rest-storage-v1", "google-rest-compute-v1"] }

Hmm, this exact URL is returning 404.

  • You didn't fill in some unexpected characters in the parameters

In the parameters of the ComputePeriodInstancesPeriodStartParams?

abdolence commented 10 months ago

Your Cargo.toml is fine.

I'm using service account saved as JSON. This service account should have the necessary permissions, but I'll double check.

I'm quite sure this is related to the way your app is authenticated. How did you generate that JSON file? Is there anything unusual about it?

Hmm, this exact URL is returning 404.

Yes, this should return 404 since this is only the base URL for the API.

In the parameters of the ComputePeriodInstancesPeriodStartParams?

Yes, something off in instance name for example?

abdolence commented 10 months ago

Try to check using:

gcloud auth application-default login

and compare those JSON files - one generated by gcloud tool, and another yours. PS. gcloud cli generates usually a file here: $HOME/.config/gcloud/application_default_credentials.json

abdolence commented 10 months ago

I checked even when you don't permissions you are supposed to get something like this:

ResponseError(ResponseContent { status: 403, content: "{\n  \"error\": {\n    \"code\": 403,\n    \"message\": \"Required 'compute.instances.list' permission for 'projects/'...

please check if you provided appropriate PROJECT_ID as well.

TmLev commented 10 months ago

I'm quite sure this is related to the way your app is authenticated. How did you generate that JSON file? Is there anything unusual about it?

I downloaded this JSON file from the Google Cloud UI for creating service accounts here:

image

I've been using this service account for uploading images to Cloud Storage and for getting the instance details (gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_get) -- both scenarios work just fine.

Can't spot anything unusual about it, here's its structure:

{
  "type": "service_account",
  "project_id": "<REDACTED>",
  "private_key_id": "<REDACTED>",
  "private_key": "<REDACTED>",
  "client_email": "<REDACTED>",
  "client_id": "<REDACTED>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "<REDACTED>",
  "universe_domain": "googleapis.com"
}

Yes, something off in instance name for example?

The same name most definitely works for ComputePeriodInstancesPeriodGetParams, so seems like everything is okay here.

and compare those JSON files - one generated by gcloud tool, and another yours.

gcloud CLI generates a different JSON:

{
  "client_id": "<REDACTED>",
  "client_secret": "<REDACTED>",
  "quota_project_id": "<REDACTED>",
  "refresh_token": "<REDACTED>",
  "type": "authorized_user"
}

please check if you provided appropriate PROJECT_ID as well.

The same project id works for ComputePeriodInstancesPeriodGetParams, I even put all three parameters in const definitions just to be sure.

TmLev commented 10 months ago

Here's the initialization bit:

    let google_rest_client = Arc::new(
        gcloud_sdk::GoogleRestApi::with_token_source(
            TokenSourceType::File("./google-service-account.json".into()),
            GCP_DEFAULT_SCOPES.clone(),
        )
        .await?,
    );

I assume GCP_DEFAULT_SCOPES should work since the error message differs from the one you get when you lack permissions?..

abdolence commented 10 months ago

This is interesting, so you're saying the same setup for the same project ID and instance names you have actually other methods working and only compute_instances_start fails? Then I was wrong and it is not related to authentication. I didn't know that it works for other methods before.

abdolence commented 10 months ago

let compute_config = google_rest_client.create_google_compute_v1_config().await

Can you elaborate how are using those configs? Are you storing them for long time and reusing? If so, please create them for each requests separately, since they contain short lived tokens.

abdolence commented 10 months ago

I tested this example with my service account and test project:

let google_project_id = gcloud_sdk::GoogleEnvironment::detect_google_project_id().await
        .expect("No Google Project ID detected. Please specify it explicitly using env variable: PROJECT_ID");

    let google_rest_client = gcloud_sdk::GoogleRestApi::new().await.unwrap();
    let compute_config = google_rest_client.create_google_compute_v1_config().await.unwrap();

    let response = gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_list(
        &compute_config,
        gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodListParams {
            project: google_project_id.to_string(),
            zone: "europe-north1-a".to_string(),
            ..Default::default()
        }
    ).await.unwrap();

    println!("{:?}", response.items.map(|xs| xs.iter().map(|x| x.name.clone()).collect::<Vec<_>>()));

    let request = gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodStartParams {
        project: google_project_id.to_string(),
        instance: "lb-mini-balancer-node".into(),
        zone: "europe-north1-a".into(),
        ..Default::default()
    };
    let response =
        gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_start(
            &compute_config,
            request,
        )
            .await.unwrap();

    println!("{:?}", response);

    let request = gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodStopParams {
        project: google_project_id.to_string(),
        instance: "lb-mini-balancer-node".into(),
        zone: "europe-north1-a".into(),
        ..Default::default()
    };
    let response =
        gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_stop(
            &compute_config,
            request,
        )
            .await.unwrap();

    println!("{:?}", response);

and it works. Something different with our environments or parameters.

TmLev commented 10 months ago

This is interesting, so you're saying the same setup for the same project ID and instance names you have actually other methods working and only compute_instances_start fails?

If we're talking about compute.instances API, I haven't tested any "mutating" requests (meaning non-GET) apart from the compute.instances.start. But yeah, GET requests work.

Can you elaborate how are using those configs?

I'm creating a new config before every request and I'm not storing them anywhere.

and it works

I will triple check everything and will try a different service account

abdolence commented 10 months ago

I think service accounts are irrelevant now - if at least one method is working. Something with either network/proxy or your parameters (structure fields for params).

TmLev commented 9 months ago

Something with either network/proxy

I've tried sending the same request from a Google Cloud instance located in the same zone/region, but it returned the same response.

Something with ... your parameters (structure fields for params).

They are the same as the ones I use for ComputePeriodInstancesPeriodGetParams.

abdolence commented 9 months ago

It is hard me to help here since this is not reproducible on my infrastructure. As one possible option to debug you can try to check HTTP request printing it out before it sent to Google. Just add some print inside locally cloned in google cloud sdk crate when the request is built. It may show something unexpected. One more option is to try to use some kind of HTTP to HTTPS simple proxy and check the traffic, but it maybe more complicated.

TmLev commented 6 months ago

I took another try at this.

I forked this repo, added it via path = "..." to Cargo.toml and made the following changes:

diff --git a/gcloud-sdk/src/rest_apis/google_rest_apis/compute_v1/apis/instances_api.rs b/gcloud-sdk/src/rest_apis/google_rest_apis/compute_v1/apis/instances_api.rs
index 989d988a2..2d4833810 100644
--- a/gcloud-sdk/src/rest_apis/google_rest_apis/compute_v1/apis/instances_api.rs
+++ b/gcloud-sdk/src/rest_apis/google_rest_apis/compute_v1/apis/instances_api.rs
@@ -7023,6 +7023,7 @@ pub async fn compute_instances_start(
         local_var_req_builder = local_var_req_builder.bearer_auth(local_var_token.to_owned());
     };

+    local_var_req_builder = local_var_req_builder.header(reqwest::header::CONTENT_LENGTH, 0);
     let local_var_req = local_var_req_builder.build()?;
     let local_var_resp = local_var_client.execute(local_var_req).await?;

It works with the explicit CONTENT_LENGTH set to 0 and fails without it. Just to be clear, I made zero changes to the code from OP or to the service account/auth stuff.

I really don't know why it works on your infrastructure.

TmLev commented 6 months ago

This may be relevant: https://github.com/seanmonstar/reqwest/issues/838

TmLev commented 6 months ago

I can also confirm that you can replace the default client of GoogleRestApi with your own, setting CONTENT_LENGTH to 0 by default for all request headers:

let mut headers = reqwest::header::HeaderMap::new();
headers.insert(CONTENT_LENGTH, 0.into());
let http_client = reqwest::ClientBuilder::new()
    .default_headers(headers)
    .build()
    .unwrap();

let google_rest_client = gcloud_sdk::GoogleRestApi::with_client_token_source(
    http_client,
    token_source_type,
    token_scopes,
).await?;

Although I'm not sure whether it will break requests with non-empty body...

abdolence commented 6 months ago

Hey, hm, interesting.

Maybe this is related that I had different reqwest version when I tested it. Right now I have "0.11.20" in my lock file. Which one do you have in yours?

abdolence commented 6 months ago

The problem with this it can be 0 only when body is actually empty. And that's not true for all requests, but only for some of them.

TmLev commented 6 months ago

Maybe this is related that I had different reqwest version when I tested it. Right now I have "0.11.20" in my lock file. Which one do you have in yours?

Same:

[[package]]
name = "reqwest"
version = "0.11.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e9ad3fe7488d7e34558a2033d45a0c90b72d97b4f80705666fea71472e2e6a1"
abdolence commented 6 months ago

I'll test it again and come back to you. Thanks for the update!

TmLev commented 6 months ago

The problem with this it can be 0 only when body is actually empty. And that's not true for all requests, but only for some of them.

I assume setting the body explicitly results in implicit update of CONTENT_LENGTH header by reqwest itself (don't quote me on that)

abdolence commented 6 months ago

I just tested it again, and it still works for me with no issues 🤔 I even added some println!() to print out whole content of that local_var_req.

You sure you don't have any proxy between your application and Google Cloud?

!!!!! Request { method: POST, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("compute.googleapis.com")), port: None, path: "/compute/v1/projects/latestbit/zones/europe-north1-a/instances/lb-mini-balancer-node/start", query: None, fragment: None }, headers: {"user-agent": "gcloud-sdk-rs/v0.24.6", "authorization": Sensitive, "authorization": Sensitive} }

Response:
Operation { client_operation_id: None, creation_timestamp: None, description: None, end_time: Some("2024-05-21T09:27:12.162-07:00"), error: None, http_error_message: None, http_error_status_code: None, id: Some("1961906089995892111"), insert_time: Some("2024-05-21T09:27:12.156-07:00"), instances_bulk_insert_operation_metadata: None, kind: Some("compute#operation"), name: Some("operation-1716308831374-618f94a380c9b-bd60f52f-e5366792"), operation_group_id: None, operation_type: Some("start"), progress: Some(100), region: None, self_link: Some("https://www.googleapis.com/compute/v1/projects/latestbit/zones/europe-north1-a/operations/operation-1716308831374-618f94a380c9b-bd60f52f-e5366792"), set_common_instance_metadata_operation_metadata: None, start_time: Some("2024-05-21T09:27:12.162-07:00"), status: Some(Done), status_message: None, target_id: Some("8330157900564331422"), target_link: Some("https://www.googleapis.com/compute/v1/projects/latestbit/zones/europe-north1-a/instances/lb-mini-balancer-node"), ...}
abdolence commented 6 months ago

Also tested it with the latest 0.11.27 with the same results.

abdolence commented 6 months ago

Another theory, maybe this is also different in different GCP regions. Which GCP region/zone are you trying to work with?

abdolence commented 6 months ago

Which OS are you using as well?

TmLev commented 6 months ago

You sure you don't have any proxy between your application and Google Cloud?

I'm sure I don't -- I even tried to start an instance in the same zone and send a request from there but it still didn't work

Which GCP region/zone are you trying to work with?

us-central1-a

Which OS are you using as well?

Locally macOS 13.4. For production, I deploy via Docker with debian:bookworm-slim as the base image.

abdolence commented 6 months ago

Just tested it on Mac OS + us-central1. Works again for me :/

This is frustrating. I tested this on different laptops now, and I just update the example gcs-rest-client with just this code:

    let compute_config = google_rest_client.create_google_compute_v1_config().await.unwrap();
    let request = gcloud_sdk::google_rest_apis::compute_v1::instances_api::ComputePeriodInstancesPeriodStartParams {
        project: google_project_id.to_string(),
        instance: "abd-test-micro".into(),
        zone: "us-central1-a".into(),
        ..Default::default()
    };
    let response =
        gcloud_sdk::google_rest_apis::compute_v1::instances_api::compute_instances_start(
            &compute_config,
            request,
        )
            .await.unwrap();

and it works without any issue.

Do you a complete simple example that doesn't work for you? (obviously, don't include any sensitive info there, like service accounts etc).