New resource of type "http"

[dduportal]

Is your feature request related to a problem? Please describe.

I'm trying to build an updatecli workflow that retrieve the maven latest version, and I would like to add a condition that checks if the tarball of the latest version exists (same idea as checking if a Docker image is published but it's a file at a given URL).

There are no way to do it, except using a condition of type file and a transformer that empties the buffer but it's hacky AND slow.

Describe the solution you'd like

I would like a new resource of type http that would allow the following:

• Source: Retrieve the body or the header of an URL
• Condition: Check if an URL exists (e.g. answer a certain type of HTTP code that could be defined) or if it has a given header defined (key or value)
• Target: POST/PUT the source to change an URL

Describe alternatives you've considered I could use the "shell" (#264) resource with curl orwget

Additional context

[olblak]

That's indeed a good suggestion

[olblak]

I think @mavimo was interested to have a look to this one. Instead of having a new HTTP plugin, we could also reuse the file plugin and just check if a http query return 200

[dduportal]

I think @mavimo was interested to have a look to this one. Instead of having a new HTTP plugin, we could also reuse the file plugin and just check if a http query return 200

Using the file resource for http feels weird : its goal is to retrieve content of a file, while http can (should?) be specified for a custom verb, a map of headers, and its return as a source could be the body or a header for instance : different use cases

[olblak]

@dduportal Good point

[dduportal]

@dduportal Good point

I’m highly motivated to start a new resource !

[dduportal]

UX Proposal for this new resource:

name: End to end test of the 'http' resource kind
pipelineid: "e2e/http"

# Sources of type http returns either the body or a header
sources:
  # Returns the content of the 'maven-metadata.xml' file content as source (multi-line, direct HTTP/200)
  getJenkinsWarArtifactMetadatas:
    kind: http
    spec:
      url: https://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/maven-metadata.xml
  # Returns the content of the 'checksums.txt' file content as source (multi-line, following redirections HTTP/3xx)
  getUpdatecli0.65.1Checksums:
    kind: http
    spec:
      url: https://github.com/updatecli/updatecli/releases/download/v0.65.1/checksums.txt # HTTP/302 (GitHub redirection to raw.githubusercontent.com)
  ## Source fails due to HTTP/3xx redirect disabled by user
  # getUpdatecli0.65.1Checksums:
  #   kind: http
  #   spec:
  #     url: https://github.com/updatecli/updatecli/releases/download/v0.65.1/checksums.txt # HTTP/302 (GitHub redirection to raw.githubusercontent.com)
  #     followredirects: false # Default is true
  # Returns the content of the Header 'Location' (e.g. the target of the HTTP redirect)
  getRedirectLocationForUpdatecliLinuxAmd64Archive:
    kind: http
    spec:
      url: https://github.com/updatecli/updatecli/releases/download/v0.65.1/updatecli_Linux_arm64.tar.gz
      response:
        header: Location
  ## Source fails with an error: the URL returns HTTP/404 (same for server-side errors such as HTTP/5xx)
  # failOnHttpError:
  #   kind: http
  #   spec:
  #     url: https://do.not.exists.com/unknown_resource
  # Returns the content of the 'maven-metadata.xml' file from the private URL (custom headers for the request)
  getFromPrivateArtifact:
    kind: http
    spec:
      url: https://private.maven.repo/releases/org/jenkins-ci/main/jenkins-war/maven-metadata.xml
      request:
        headers:
          Authorization: 'Bearer Xcjkdhvcjidcdscdcplcmdz'
          Accept: 'application/xml'

conditions:
  # Returns 'true' if the specified URL returns HTTP/1xx, HTTP/2xx or HTTP/3xx
  checkForURL:
    kind: http
    spec:
      url: https://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/maven-metadata.xml
  ## Conditions returns 'false' as the URL returns HTTP/404
  ## If there is an HTTP/500 (server side error) then the conditions fails with an ERROR (different than returning false which "skips" the pipeline)
  # checkForNonExistingUrl:
  #   kind: http
  #   spec:
  #     url: https://do.not.exists.com/unknown_resource
  # Returns 'true' as the URL exists
  checkForURL:
    kind: http
    spec:
      url: https://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/maven-metadata.xml
  # Returns 'true' if the specified URL returns HTTP/1xx, HTTP/2xx or HTTP/3xx to the custom request (custom verb and headers)
  checkWithCustomRequest:
    kind: http
    spec:
      url: https://private.maven.repo/releases/org/jenkins-ci/main/jenkins-war/maven-metadata.xml
      request:
        verb: HEAD
        headers:
          Authorization: 'Bearer Xcjkdhvcjidcdscdcplcmdz'
  # Returns 'true' if the response header 'Location' contains the same value as the source 'remoteUrl'
  getRedirectLocationForUpdatecliLinuxAmd64Archive:
    kind: http
    sourceid: remoteUrl
    spec:
      url: https://github.com/updatecli/updatecli/releases/download/v0.65.1/updatecli_Linux_arm64.tar.gz
      response:
        header: Location
  # Returns 'true' if the response header 'Location' contains the value 'https://google.com'
  getRedirectLocationForUpdatecliLinuxAmd64Archive:
    kind: http
    disablesourceinput: true
    spec:
      url: https://github.com/updatecli/updatecli/releases/download/v0.65.1/updatecli_Linux_arm64.tar.gz
      response:
        header: Location
      assertvalue: https://google.com
  # Returns 'true' if the response code is HTTP/302
  getUpdatecli0.65.1Checksums:
    kind: http
    spec:
      url: https://github.com/updatecli/updatecli/releases/download/v0.65.1/checksums.txt # HTTP/302 (GitHub redirection to raw.githubusercontent.com)
      response:
        code: 302

# No target

=> @mavimo @olblak would this proposal cover your use cases? => For the jenkins-infra project we would benefit of this resource for the following use cases:

• Checking for package availabilities:
  • chocolatey (for instance https://github.com/jenkins-infra/packer-images/blob/1255193338032ef4439022f2e954ba75eb5f00df/updatecli/updatecli.d/trivy.yaml#L35)
  • Checking for Maven binary on download sites (for instance https://github.com/jenkins-infra/packer-images/blob/1255193338032ef4439022f2e954ba75eb5f00df/updatecli/updatecli.d/maven.yml#L37)
  • Checking for NodeJS binary (for instance https://github.com/jenkins-infra/packer-images/blob/1255193338032ef4439022f2e954ba75eb5f00df/updatecli/updatecli.d/nodejs.yml#L36)
• Retrieving checksum files to verifiy downloads (as we tend to have GPG keys in our repositories)

[olblak]

@dduportal Your proposal is very nice and would be interesting. In my case, I was looking for something simpler :D just checking that a file exsit on a scm repository :p