Issue when deploying through Intellij on version 22.11.1-221

[cgatlin00]

Hello, I have upgraded to version 22.11.1-221 of Cloud Code Plugin within IntelliJ and since that upgrade I haven't been able to deploy my application when running on v2.0.2 of skaffold and the new version of Cloud Code. If I deploy through skaffold via the command line using the following: skaffold run --port-forward --tail --cleanup=false --status-check=false it runs just fine and my pods come up normally. If I attempt to deploy via IntelliJ I get an issue with my redis instance as it seems IntelliJ is adding the following label: ide: idea and I get the following error message when looking at my redis instance: message: 'Update call failed: error applying desired state: summary: Error creating Instance: googleapi: Error 400: resource labels are invalid: key "ideVersion" contains invalid character ''V'' at index 2' My team is wanting to upgrade to the new version of cloud code and skaffold, but until this issue is resolved this is preventing us from doing that.

[etanshaul]

hi. are you pointing Cloud Code to an locally installed version of skaffold (v2.0.2) (vs. using the managed / bundled version)?

If you use the bundled version of skaffold does it work?

We'll check on this issue, but we were planning on bundling 2.0.2 with Cloud Code in the next release. Are there features in 2.0.2 that you are particularly interested in? edit - I totally understand wanting to use the latest version of skaffold; I'm just trying to understand the severity of how blocked you are to help with our triage of this issue.

[cgatlin00]

Hey @etanshaul thank you for the reply. I don't believe I am pointing to a locally installed version of skaffold. I can downgrade my local version to v2.0.0 and give that a try. But from my understanding Cloud Code plugin within IntelliJ should run separate from my local version of skaffold is that correct? The main reason I have a local version was to test my changes to see if it was specific to IntelliJ or not.

[etanshaul]

But from my understanding Cloud Code plugin within IntelliJ should run separate from my local version of skaffold is that correct?

Yes that is correct. By default Cloud Code bundles its own version of skaffold, but you can override this under Settings > Tools > Cloud Code > Dependencies:

It appears as though the bundled version is v2.0.1

[etanshaul]

Another thing to check. When you first run the kubernetes session, something like the following should be printed out:

/Users/.../Library/Application Support/cloud-code/bin/versions/6b5b8a549019ab635d48b30614e5422807698685b633020f5fb2f1901b04814c/skaffold dev --filename skaffold.yaml --label ide=idea --label ideVersion=2022.2.4.0.0 --label ijPluginVersion=22.11.1-222 --auto-build=false --auto-deploy=false --auto-sync=false --default-repo  --rpc-port 50051 --port-forward=true --wait-for-deletions-max=2m0s --status-check=true
failed to detect active kubernetes cluster node platform. Specify the correct build platform in the `skaffold.yaml` file or using the `--platform` flag
Listing files to watch...

Notice that I also have --label ideVersion=2022.2.4.0.0. If you could share a snippet of your output as well that could help.

EDIT

I just noticed that your error message: 'Update call failed: error applying desired state: summary: Error creating Instance: googleapi ... is actually coming from applying this label, not from skaffold rejecting the flag.. interesting. Investigating possible causes here..

[cgatlin00]

Yeah that is the interesting thing. So when running my application within IntelliJ I get that output you just mentioned. When running the skaffold command at the command line it seems to run just fine and all of my pods come up normally. This seems interesting to me.... invalid: key "ideVersion" contains invalid character ''V'' at index 2'

[etanshaul]

yeah. I just tried with Cloud Code and skaffold v2.0.2 and it worked. So I don't think it is an issue with the version of skaffold. Rather, it looks to be your cluster rejecting this key.

Do you know if this is a new issue (was it working before)?

[etanshaul]

When running the skaffold command at the command line it seems to run just fine and all of my pods come up normally

Right, but are you invoking skaffold exactly the same way as Cloud Code is (with the same flags, including all the labels)? (cloud code prints out its skaffold invocation in the output, so you can copy that and try directly).

[cgatlin00]

Alright I tried what you suggested, this was the output from Cloud Code: skaffold dev --filename skaffold.yaml --label ide=idea --label ideVersion=2021.3.3.0.0 --label ijPluginVersion=22.11.1-221 --auto-build=false --auto-deploy=false --auto-sync=false --rpc-port 50051 --port-forward=true --wait-for-deletions-max=2m0s --status-check=true --verbosity warn and this was the command I have been using skaffold run --port-forward --tail --cleanup=false --status-check=false.

The second command works just fine, the first does not and I get the same error as before. So I assume it is one of the flags being set by Cloud Code. Is there a way to manually override this?

[etanshaul]

Is there a way to manually override this?

No not at the moment unfortunately. There are a couple "hard coded" flags that Cloud Code is adding. We can consider removing / making this togglable, however, the better approach might be to first figure out the root cause here.

To ask again - do you know if this is a new issue - was Cloud Code working for you at some point earlier?

[cgatlin00]

To ask again - do you know if this is a new issue - was Cloud Code working for you at some point earlier?

Sorry I thought I had answered this question already. Yes, my entire company and team rely on Cloud Code and have been using it since the start of this year. But we are attempting to upgrade to version v2.0.1 and since attempting to upgrade this is where we have started to experience problems. Others on my team still running on version v1.39.4 it works just fine for them.

[etanshaul]

thanks. thats helpful. Checking here if we can figure out a cause.

[cgatlin00]

Sure thank you very much, appreciate the help!

[etanshaul]

by the way, are you using GKE?

maybe you could try to directly apply a label to one of your resources matching the failed one to see if that is rejected. I.e. https://kubernetes.io/docs/reference/kubectl/cheatsheet/#updating-resources (using kubectl label)

The key/values indeed look permissible per https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set

[cgatlin00]

Yes we are using gke, and I will give that a try and see if that makes any difference.

[tejal29]

Thanks @cgatlin00 for reporting this error. @etanshaul is there a way we could ask @cgatlin00 to get debug logs for their run. We will want them to pass the -v=debug flag to skaffold process.

[etanshaul]

yes! you can configure this in the run configuration:

Under "advanced settings" set the logging verbosity to DEBUG:

[cgatlin00]

So I think I have gotten a little bit more information as to why it is failing. In the skaffold command produced by the Google Cloud plugin, it sets the following label: ideVersion=2021.3.3.0.0. When divining into our redisInstance that specific label is producing the original error that I posted. If I change it to ideversion with a lowercase v that resolved the invalid character, but still get an error as the value of 2021.3.3.0.0 contains "." which are against syntax.

When looking here that specifies labeling conventions and the label above does not follow those conventions. Is this something you guys will be able to fix? I believe the same is true for the ijPluginVersion=22.11.1-221 label.

[etanshaul]

ah yes:

Keys and values can contain only lowercase letters, numeric characters, underscores, and dashes. All characters must use UTF-8 encoding, and can include international characters.

so there is a disconnect between gcp resources vs kubernetes resources in terms of allowable characters.

@tejal29 should those label fields supplied to skaffold only be applied to kubernetes resources?

[etanshaul]

also - in order not to stall to much on this, we'll update them to be lowercase. This should go out in the next insider (https://cloud.google.com/code/docs/intellij/insiders) release this week, and to stable in a few weeks.

[cgatlin00]

Okay sounds great! I think it is also causing an issue with the periods in the label. Because when removing the capital V it then threw an error with the periods between 2021.3.3.0.0.

[etanshaul]

I think it is also causing an issue with the periods in the label. Because when removing the capital V it then threw an error with the periods between 2021.3.3.0.0.

Good to know. we'll clean this up then. and potentially remove them if they are not important.

[tejal29]

@tejal29 should those label fields supplied to skaffold only be applied to kubernetes resources?

No. At skaffold's level, we don't do any validation on the label and key value for exactly the above reason. It could differ between cluster provider. The same resource could be deployed to GKE or local minikube. Skaffold applies the label to all resource types (allowlisted) irrespective of where they are deployed.

I am surprised, GCP has different validation rules.

[cgatlin00]

Thank you for the information and the quick responses. I think it would maybe be helpful if these options were able to be manually selected by the user, and maybe something that could be implemented at somepoint in the future!

[etanshaul]

Thank you for the information and the quick responses. I think it would maybe be helpful if these options were able to be manually selected by the user, and maybe something that could be implemented at somepoint in the future!

I think removing them would equally work for you? Unless you are asking for the ability to apply arbitrary labels yourself to the skaffold invocation?

[cgatlin00]

No I think removing them would work as well! Just was thinking of giving users more control in the future, but removing them totally works for us! Can you let me know when those changes are implemented and in which version?

[etanshaul]

Can you let me know when those changes are implemented and in which version?

definitely. will update the thread when it is available.

[cgatlin00]

Hey one other somewhat related question, do you plan to support api version: skaffold/v4beta1 in the near future?

[cgatlin00]

Hey @etanshaul just wanted to see if you had any update on timeframe for these changes?

[ruomengz]

Hey @cgatlin00 , this issue is fixed in the latest 22.12.1 version.

[etanshaul]

Hey one other somewhat related question, do you plan to support api version: skaffold/v4beta1 in the near future?

Yes we are looking into this. No immediate timeline yet, but likely the next stable release.

Let us know if you still run into any issues with the label fix. Thanks.