ollik1 commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Cloud Composer supports installing PyPI packages from private repositories. In addition, the TF provider supports installing packages from public PyPI. However, it seems adding the required pip.conf file is not possible in TF so private repositories can't be used at the moment.

New or Affected Resource(s)

google_composer_environment

Potential Terraform Configuration

resource "google_composer_environment" "test" {
  name   = "mycomposer"
  region = "us-central1"

  config {
    software_config {

      pypi_packages = {
        numpy = ""
        scipy = "==1.1.0"
        myprivatepackage = ""  # This package is hosted in https://my-example-private-repo.com/
      }

      # This is the suggested addition
      pip_conf = <<EOT
[global]
extra-index-url=https://my-example-private-repo.com/
EOT
    }
  }
}

References

0000

b/302797531

handol-park commented 2 years ago

Anyone has a way to work around this?

CaliWee commented 1 year ago

I haven't implemented it yet, but I think you can specify an object in the airflow bucket to configure pip: https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#install-private-repo https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_object

Gilles-Marco commented 1 year ago

I haven't implemented it yet, but I think you can specify an object in the airflow bucket configure pip: https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies#install-private-repo https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_object

But you only know the bucket after composer is created. Composer need this pip.conf to install lib but this pip.conf need the composer to be created to have the bucket.

So A is dependant of B but B is also dependant of A

There was a suggestion to decouple google_composer_environment and python dependencies https://github.com/hashicorp/terraform-provider-google/issues/12781

It could be a fix to this issue

CaliWee commented 1 year ago

@Gilles-Marco You're right. In order to use the approach I mentioned before, I had to deploy the composer module first, then add the object to the configuration and deploy again... not ideal. Put a vote down on the suggestion you linked.

CaliWee commented 1 year ago

One way that this might be enabled is if the composer module could take an existing GCS bucket as a parameter. Then the bucket could be pre-populated with a pip.conf which would configure the private repository

guicane commented 11 months ago

Another comment and vote to have this issue resolved.

It is annoying and not ideal. I'm struggling to get this working.

I'm not sure this is a new change to the API but when reapplying the terraform configuration after adding the pip.conf with the correct internal package repo to the bucket, the google api returns an error saying it can't destroy due to it not finding the resource URL. Anyone had this issue? Seems that it adds the v1beta1/ to the url and it can't find it. This was working before and it has stopped recently.

Error: googleapi: got HTTP response code 404 with body: <!DOCTYPE html>
│ <html lang=en>
│   <meta charset=utf-8>
│   <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
│   <title>Error 404 (Not Found)!!1</title>
│   <style>
│     *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
│   </style>
│   <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
│   <p><b>404.</b> <ins>That’s an error.</ins>
│   <p>The requested URL <code>/v1beta1/projects/project_id_here/locations/europe-west2/environments/?alt=json&amp;prettyPrint=false</code> was not found on this server.  <ins>That’s all we know.</ins>

If this was solved as per suggestion above this wouldn't even be an issue.

CaliWee commented 11 months ago

@guicane Whats you terraform configuration look like? Just the composer / pip related stuff + provider block

guicane commented 11 months ago

@guicane Whats you terraform configuration look like? Just the composer / pip related stuff + provider block

I'm using the official Google TF module to create composer. I added more details and debug logs to the issue I created about this same error.

16540

This error only comes up when redeploying composer after adding pip.conf to the composer bucket. If we could place a pip.conf file inside during the deploy none of this would happen. :(

CaliWee commented 10 months ago

This ticket is now solved by: https://github.com/hashicorp/terraform-provider-google/issues/14586

You can now provide a storage config to your Composer environment where you specify an existing GCS bucket. In this bucket you can save a pip.conf, configuring access to your private PyPi. As long as the Service account building the composer environment is authorized to access the private PyPi, it will successfully install private python packages when it is created. Examples provided in the linked issue

hashicorp / terraform-provider-google

Support private PyPI repositories #10850

Community Note

Description

New or Affected Resource(s)

Potential Terraform Configuration

References

0000

16540