[BUG] API not present in configuration file is still being deployed

fabienmazieres commented 2 months ago

Release version

v6.0.11

Describe the bug

As per documentation and #435, I was told that I should be able to prevent the deployment of an API by omitting the API from the configuration file.

However, this does not appear to work as I have an API currently commented in configuration code:

[...]
apis:
[...]
  # - name: api2
  #   properties:
  #     serviceUrl: ""
  # there is no dev env

but the API is being deployed whether I make a publish all artifacts or edit the specs and publish last commit... I always deploy with configuration file as I have multiple environment, I am using different configuration files for each environment

Expected behavior

Only API defined explicitly in the configuration files are published. API that are commented should not be published.

Actual behavior

All API are deployed, all the time.

Reproduction Steps

configuration file configuration.d1.yaml:

---
apimServiceName: apimdev01
apis:
#APIs Begin
  - name: api1
    properties:
      serviceUrl: "https://api1.example.net"
  # - name: api2
  #   properties:
  #     serviceUrl: ""
  # there is no dev env

folder structure:

apis
- api1
  - apiInformation.json
  - specification.json
- api2
  - apiInformation.json
  - specification.json

github-actions[bot] commented 2 months ago

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

fabienmazieres commented 2 months ago

Based on what I can see when I debug the publisher ("Launch Publisher" in VS code), it seems that there is a part of the code that look at files on the disk and will always push config that is stored locally, regardless if there is an entry in the configuration file, there does not seem to be a filter to look of files on disk AND match an entry in the configuration file

guythetechie commented 2 months ago

@fabienmazieres - are you passing the full path of configurationd1.yaml to the publisher? You can see how it's defined in our starter ADO and GitHub pipelines.

fabienmazieres commented 2 months ago

@guythetechie , I am following the GitHub pipeline example:

          CONFIGURATION_YAML_PATH: ${{ GITHUB.WORKSPACE }}/${{ inputs.CONFIGURATION_YAML_PATH }}

I have changes a few things from the template to adapt to our authentication and self-hosted runners but this part of run-publisher-with-env.yaml was left as per the original file, I did copy/paste code from your repo when working on upgrade to 6.0

guythetechie commented 2 months ago

@fabienmazieres - after rereading your issue, I think there's a misunderstanding of how the extractor and publisher work with configuration.

The extractor configuration file specifies which resources should be extracted. If resourceType: [resource1, resource2, ...] is specified, when the extractor extracts resources of resourceType, only resource1 and resource2 will be extracted. If nothing is specified, all resources will be extracted.
The publisher configuration file allows you to override the properties of resources that are published. It does not influence which resources get published. The publisher chooses resources to publish like this:
- If a commit ID is passed in configuration, only resources changed in that commit will be published.
- If no commit ID is passed in configuration, all artifacts will be published.

There is no built-in functionality to only publish (or exclude from publishing) a specific resource. You can only control what gets extracted. Once something lands in your artifacts source control, it's eligible for publishing.

fabienmazieres commented 2 months ago

@guythetechie, Could it be worth to have as a feature the capability to publish only the specified API (or other component of the APIM config)?

guythetechie commented 1 month ago

@fabienmazieres - I don't think so. It's not a simple task, and it goes against our philosophy of Git being the source of truth.

If different teams are responsible for different components of the same APIM instance, they can work with different repositories. Team A can work in repository A where only artifacts related to API A are extracted. They have their own extractor/publisher pipeline. Similarly, team B can work in repository B with their separate artifacts and extractor/publisher pipelines. Both teams can target the same APIM instance, and only their relevant resources will be published.

Azure / apiops