Make upstream is not added added to bridged provider templates.

VenelinMartinov commented 10 months ago

Make upstream is not called during tests in GHA for bridged providers. It is a noop for non patched providers, so should be fine to call.

Some patched providers don't test patched upstreams in non-integration tests. Example: https://github.com/pulumi/pulumi-cloudflare/blob/master/.github/workflows/run-acceptance-tests.yml

It doesn't have any non-integration test, so it doesn't really affect us but it might catch someone off guard like it did me here https://github.com/pulumi/pulumi-azuredevops/pull/219

Native Providers

[ ] aws-apigateway
[ ] aws-native
[ ] awsx
[ ] azure-native
[ ] command
[ ] eks
[ ] google-native
[ ] kubernetes

Bridged Providers

Tier 1

[ ] aws
[ ] gcp
[ ] azure
[ ] azuread

Tier 2

[ ] aiven
[ ] akamai
[ ] alicloud
[ ] archive
[ ] artifactory
[ ] auth0
[ ] azuredevops
[ ] civo
[ ] cloudamqp
[ ] cloudflare
[ ] cloudinit
[ ] confluentcloud
[ ] consul
[ ] databricks
[ ] datadog
[ ] digitalocean
[ ] dnsimple
[ ] docker
[ ] ec
[ ] external
[ ] f5bigip
[ ] fastly
[ ] github
[ ] gitlab
[ ] hcloud
[ ] http
[ ] kafka
[ ] keycloak
[ ] kong
[ ] libvirt
[ ] linode
[ ] local
[ ] mailgun
[ ] minio
[ ] mongodbatlas
[ ] mysql
[ ] newrelic
[ ] nomad
[ ] ns1
[ ] null
[ ] oci
[ ] okta
[ ] onelogin
[ ] openstack
[ ] opsgenie
[ ] pagerduty
[ ] postgresql
[ ] rabbitmq
[ ] rancher2
[ ] random
[ ] rke
[ ] signalfx
[ ] slack
[ ] snowflake
[ ] splunk
[ ] spotinst
[ ] sumologic
[ ] tailscale
[ ] tls
[ ] vault
[ ] venafi
[ ] vsphere
[ ] wavefront

t0yv0 commented 10 months ago

https://github.com/pulumi/pulumi-cloudflare/blob/master/Makefile#L120 CLoudflare calls make provider which does call the patching via Makefile. Likewise AWS is a bridged provider and certainly does patch it before building. I'm not sure this is accurate? do you have a repro?

VenelinMartinov commented 10 months ago

Yeah, I think the issue only applies for tests which don't use the provider executable but instead build against provider code, like the GRPC replay tests.

For example in https://github.com/pulumi/pulumi-azuredevops/pull/219 I added GRPC tests but the CI was failing since the tests were building against a non-patched upstream.

I doubt there is any code which is wrong in the wild.

t0yv0 commented 10 months ago

I'm not sure this makes any sense still

https://github.com/pulumi/pulumi-aws/actions/runs/7142409416/job/19451648033#step:11:1

prerequisites job runs

make provider # this implies make upstream
make test_provider

t0yv0 commented 10 months ago

Looking at the azdo PR.

t0yv0 commented 10 months ago

Ah I see the workaround is adding make upstream to preTest

  preTest:
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-region: ${{ env.AWS_REGION }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        role-duration-seconds: 3600
        role-session-name: ${{ env.PROVIDER }}@githubActions
        role-to-assume: ${{ secrets.AWS_CI_ROLE_ARN }}
    - name: Make upstream
      run: make upstream

Providers that do that are not affected. This makes sense.

pulumi / ci-mgmt