Use Windows Certificate Store or add command line options for Server CA certs and allow tasks to accept command line arguments for the JF CLI

fourpastmidnight commented 1 year ago

Is your feature request related to a problem? Please describe.

Yes. Your "solution" for when organizations use a custom CA is ridiculous. In my organization, I do not have access to the build servers. I have a hacky solution for this right now that is FAR from ideal. Also note that my HACKY solution, thus far, is not working. My build failed when pushing packages to Artifactory.

During the build, my HACKY soluction successfully created the PEM certificate files in the specified location, $(Agent.ToolsDir)/_jf/security/certs and named the files custom.crt. I assume that since the build failed to push the packages due to not being able to establish a TLS connection because JF CLI could not validate the server certificate means that the custom.crt file was never even used.

[!NOTE] Please note the following:

I assume the certificate file needs to be in PEM format (as opposed to PKCS12 (e.g., .p12)) EDIT: This assumption is correct.

I verified the file custom.crt exists at the expected location

I verified that custom.crt entire certificate chain of trust is valid, as it contains the Artifactory server certificate as well as the intermediate and root CA certificates in the chain of trust.

That the README doesn't say that the certificate has to have any particular name

That the README says to check the JF CLI documentation but that documentation ALSO makes no mention of this requirement!!!

In short, you made this TOO HARD to use when using an internal PKI and with strict access controls to the build environment, as many larger organizations do!

Describe the solution you'd like to see

I'd like to see one or both of the following features:

Use the Windows Certificate Store when running on Windows operating systems in order to validate server certificates
- Given JF CLI is written in Go, which is a cross-platform language, I would hope that this would be possible
If option 1 is not possible, then the next two options would be desirable:
- Allow JFrog Azure DevOps tasks to accept additional command line arguments
  - What would immediately help me is the --insecure-tls option because I know my cert is just fine, it just can't be validated with whatever validation technique the JF CLI is currently using.
  - What would help me even more is the ability to specify command line options such as --ca-cert which I could then use to specify the path to a file containing the chain of trust for the server certificate for the purposes of TLS validation.

Describe alternatives you've considered

The only alternative I have to consider is using the raw Jfrog CLI task to "recreate" what the Jfrog Push Deployment Artifacts task does, only that will allow me to provide the --insecure-tls option to get around this error. But honestly, this should not be necessary.

fourpastmidnight commented 1 year ago

I took a cursory glance at the code, and it doesn't even appear that the certificate is being looked for, contrary to the README! Mind you, perhaps it's looked for somewhere else; I'll keep looking. Mainly, I'm looking to see if information was omitted from the README, like, the certificate file needs a particular name, like it does for Artifactory Server....

fourpastmidnight commented 1 year ago

OK, perusing the code some more, I don't see anywhere where an attempt to use a certificate located at $(Agent.ToolsDirectory)/_jf/security/certs is made. Maybe jf.exe does this "automagically", that's my next check, but their documentation makes no mention of supplying a certificate. It would be nice in the README, that you provided a direct link to said documentation, if it exists, because I'm not seeing it.

fourpastmidnight commented 1 year ago

Finally found the docs for usage of a private CA certificate in a non-obvious location. Again, a "deep link" would've been nice.

fourpastmidnight commented 1 year ago

Looking at the code for the JF DotNet task, for pushing packages to Artifactory, it calls a function named performDotNetNugetPush, which further calls into the JF CLI via a function utils.executeCliCommand.

For the generic task Jfrog CLI V2, that instead ends up calling a function named utils.executeCliTask.

For one thing, I can see in utils.js that executeCliTask performs the following action:

process.env.JFROG_CLI_HOME = jfrogFolderPath;

while executCliCommand (which is called by performDoNetNugetPush) does no such thing. Because JFROG_CLI_HOME is never set, the certificate will attempt to be found in ~/.jfrog, I'm assuming on Windows, this would get translated to C:\Users\<BuildAgentServiceAccountUsername>\.jfrog, but of course, that's not where I put the certificate, pursuant to the README (and JF CLI) documentation. Shouldn't all functions which execute a JF CLI command be setting JFROG_CLI_HOME?

So, one attempt I can make is to explicitly set the JFROG_CLI_HOME environment variable.

fourpastmidnight commented 1 year ago

OK, I used the Jfrog CLI V2 task item to "recreate" the Jfrog Dotnet task item for pushing packages. I tried first without any additional parameters, since that task seems to use executeCliCommand which properly sets the JFROG_CLI_HOME enviornment variable.

However, that did not work. So then I added the JFROG_CLI_HOME environment variable with the value $(Agent.ToolsDirectory)/_jf/ to my build pipeline (I tried with both a trailing / and a value without the trailing /.) No luck.

So, my Hail Mary pass, use the --insecure-tls command-line switch. That worked.

Bottom line, this is broken. I'll be filing an issue as a bug for this. But the feature request here remains: please, just use the Windows Certificate Store on Windows.... I don't see what's so hard about that.

fourpastmidnight commented 1 year ago

I filed bug #461 since none of these tasks work, despite following the README instructions about placing certificates in the $(Agent.ToolsDirectory)/_jf/security/certs folder. I also submitted a Jfrog ticket to the same effect.

fourpastmidnight commented 11 months ago

I solved my particular problem.

The main issue is with the Jfrog Artifactory documentation for setting up TLS. Not everybody knows PKI well. I only just learned it well (ok, better 😉 ) a few years ago—but apparently, I had a gap in my knowledge.

The JFrog Artifactory documentation seems to assume that, when setting up the server with TLS (not using a reverse proxy), that you have a simple, 2-layer PKI hierarchy: Endpoint-Entity → Root CA. In this extremely simple case, you put the Root CA certificate into a file named ca-certs.crt and the endpoint entity certificate (whose subject common name is the IP address of the JFrog Artifactory node) into the custom-server.crt file. And this would work just fine.

What tripped me up, is that I have (at least) a 3-layer PKI hierarchy: Endpoint-Entity → Subordinate CA → Root CA. The documentation proffers no information on how to configure JFrog Artifactory in this situation (perhaps because it is presumed that configuring this correctly ought to be known?). What I didn't understand at the time is that when a server sends its endpoint-entity certificate to a requesting browser agent, it also sends any issuing subordinate CA certificates in the trust path back to the trust anchor Root CA for the endpoint-entity. And because I didn't know this, and the CA certificate file is literally named ca-certs.crt (notice it's plural--which is fine if you have multiple Root CAs you need to have JFrog Artifactory trust), I believed that the subordinate CA certificate should also be placed in the ca-certs.crt file rather than the custom-server.crt file—and so the custom-server.crt file contained only the endpoint-entity certificate. So, while browsing the JFrog Artifactory application interface, the browser indicated that the connection is secure (On my local machine, the Subordinate CA certificate must be in my certificate store, perhaps under Intermediate Trust Authorities (on Windows)). But when the JF CLI tried to connect to JFfrog Artifactory, and because it is not using the Windows Certificate Store, when the endpoint-entity certificate was validated, the certificate chain of trust could not be completed back to a known Root CA certificate (even though the Subordinate CA certificate was in ca-certs.crt—I assume this has something to do with how this file is processed during validation. In any event, there's no problem with validation; my configuration is just wrong.)

After searching about certificate bundles, I came across a Security Stack Exchange post that helped me understand how web servers send not only the endpoint-entity certificate, but also subordinate CA certificates in the endpoint-entity certificate trust path, I realized I had mis-configured the certificate files.

After moving the Subordinate CA certificate to the custom-server.crt file and removing it from ca-certs.crt, the JFrog CLI tasks worked as intended.

I implore JFrog to please make the documentation around configuring JFrog Artifactory more clear. PKI is, unfortunately, NOT common knowledge. I took the time to learn it because I was, at one point, afraid of it to the point where it was just ridiculous—I just needed to learn it. But even so, this gap in my knowledge lead to a lot of tinkering, and had the documentation been more clear about how to configure JFrog Artifacotry, this would have been unnecessary wasted time and effort, not only by myself, but also JFrog support staff.

fourpastmidnight commented 11 months ago

Also, I should note, that after I properly configured the certificate files on the JFrog Artifactory server, I did not need to place any certificates in $JFROG_CLI_HOME/security/certs. Everything just worked out of the box when it came to communicating with JFrog Artifactory. And this is because in my organization (which uses Windows on the desktop), our Root CA certificate is, of course, distributed to all of our workstations' Windows Certificate Store. (Which also may mean that the JF CLI does now use the Windows Certificate Store????? Otherwise, this should not have worked without me at least providing the Root CA certificate in the $JFROG_CLI_HOME/security/certs folder!)

fourpastmidnight commented 5 months ago

I ran into this AGAIN today. The JFrog Tools Installer DOES NOT use the Windows Certificate Store. So, while my build server has my Artifactory server's trust anchor root certificate in the Windows Certificate Store, the JFrog Tools Installer failed to validate the certificate due to it being "self-signed" because whatever built-in certificate store this step is using, does not include our custom Root CA certificate.

This is a HUGE issue on Windows build servers. This tool MUST have an (rather, the default) option to use the Windows Certificate Store when being run on Windows, and to fallback to any other mechanism it uses in the event the Windows Certificate Store comes up empty for a trust anchor certificate.

Really, this is just ridiculous. I've wasted MANY HOURS on this issue. This should not happen--this should be easy for customers.

[!NOTE] I am in an organization where I do not control the build servers, so it's not like I can just "Drop a file" in the documented location to "make this work". This just needs to work "out of the box" on Windows using standard Windows features, such as the Windows Certificate Store.

fourpastmidnight commented 5 months ago

OK, I have more news about this. I was able to work around the issue. However, I think it's important to discuss what the REAL issue is, and also importantly, how JFrog can help others in my situation understand the problem and how to resolve it.

First, this is NOT an issue with the JFrog Azure DevOps Extension, itself, nor with the JFrog CLI. It is an environmental issue that may occur for some customers when running tasks from this extension.

What made this particular issue tricky was that my build pipeline was failing attempting to install the JFrog CLI via the JFrog Tools Installer task. I made a (bad, as it turns out) assumption that the bundled JFrog CLI was dogfooding itself to get the requested JFrog CLI version installed on the build agent. Based on that assumption, I followed the documentation for the JFrog CLI and where to place certificates for its use when using a custom Certificate Authority for securing Artifactory.

During a call with a JFrog support engineer, we perused the source code of the JFrog Tools Installer task and discovered that the task actually uses a Microsoft-provided JavaScript library to conduct the download of a new JFrog CLI version. This aligns with why following the JFrog CLI documentation regarding certificates wasn't working—the JFrog CLI was never involved in this process.

Performing a web search about why I was getting a Error: Self signed certificate in certificate chain in my build, I came across this Stack Overflow Q&A

Older Azure DevOps Build Agents come bundled with a version of Node.JS which does not use the Windows Certificate Store. In my particular case, I do not own the build servers and therefore I am not able to update the build agent. But, what can be done is place the Artifactory server certificate's custom Root CA Certificate, in PEM format, in a file on the build server, and set an environment variable so that the older version of Node.JS will pick it up and use it. The easiest way to accomplish this is to add a build pipeline variable (for classic build pipelines) or a task variable (for YAML pipelines) called NODE_EXTRA_CA_CERTS. Its value should be a path to a file containing all the extra certificates needed to validate certificate chains for server certificates that may be presented during the build process (e.g., NODE_EXTRA_CA_CERTS could point to $(Build.SourcesDirectory\build\node\certs\ca-bundle.crt). The file should be in PEM format. You can also add a PowerShell task to export the required certificates from the Windows Certificate Store, converting them to PEM, and concatenating the results into a single file pointed to by the NODE_EXTRA_CA_CERTS variable. And then the JFrog Tools Installer task works.

Here's an example PowerShell script:

function Export-PEMCertificate {
    [CmdletBinding()]
    [OutputType([string])]
    Param (
        [Parameter(Mandatory, Position = 0, ValueFromPipeline)]
        [Microsoft.CertificateServices.Commands.Certificate[]] $Certificate
    )

    Begin {
        $Bundle = @()
    }

    Process {
        @($Certificate) | ForEach-Object {
            $Bundle += "-----BEGIN CERTIFICATE-----`n{0}`n-----END CERTIFICATE-----" -f [System.Convert]::ToBase64String(
                $_.export([System.Security.Cryptography.X509Certificates.X509ContentType]::Cert), 
                'InsertLineBreaks'
            )
        }
    }

    End {
        $Bundle -join "`n"
    }
}

# Build variables will be automatically expanded in PowerShell code.
# This can make for some interesting PowerShell code, especially in classic build pipelines.
# Note that I used a build variable named NODE_EXTRA_CA_CERTS, but one could also use
# NODE.EXTRA.CA.CERTS--but the code below would need to change accordingly.
Write-Host 'Exporting certificates to ''$(NODE_EXTRA_CA_CERTS)''...'
$NodeExtraCACertsPath = Split-Path '$(NODE_EXTRA_CA_CERTS)' -Parent
if (!(Test-Path $NodeExtraCACertsPath)) {
    Write-Host "Path not found for Node.JS Extra CA Certificates: '$NodeExtraCACertsPath'."
    $null = New-Item -Type Directory -Path $NodeExtraCACertsPath -Forc
    Write-Host "Created path '$NodeEXtraCACertsPath'."
} else {
    Write-Host "Path exists for Node.JS Extra CA Certificates: '$NodeExtraCACertsPath'."
}

if (!(Test-Path '$(NODE_EXTRA_CA_CERTS)')) {
    # Obtain a certificate using Get-ChildItem 'Cert:\<Me|LocalMachine>\<CertStore>\<thumbprint>'
    # Or, use a list of thumbprints @(<list-of-thumbprints>) | ForEach-Object { Get-ChildItem 'Cert:\<Me|LocalMachine>\<CertStore>\$_' }
    # Or get all certs in a particular store: Get-ChildItem -Recurse 'Cert:\Me\LocalMachine>\<CertStore>
    # Or get ALL certificates from the Windows Certificate Store: Get-ChildItem -Path 'Cert:\' -Recurse
    # Let's assume I have a thumbprint stored in the build variable $(CA_CERT_THUMBPRINT) and that
    # the certificate is stored in the local machine's Trusted Root Certificates certificate store...
    Get-ChildItem -Path 'Cert:\LocalMachine\Root\$(CA_CERT_THUMBPRINT)' |
        Export-PEMCertificate |
        Set-Content -Path '$(NODE_EXTRA_CA_CERTS)' -Encoding ASCII

    Write-Host 'Exported required CA certificate(s) to ''$(NODE_EXTRA_CA_CERTS)''.'
} else {
    Write-Host 'The required CA certificate(s) have already been exported to ''$(NODE_EXTRA_CA_CERTS''.'
}

I'll try to submit a PR for the documentation to add this bit of information (hopefully in a more clear way than this has been presented...).

brettski commented 4 months ago

Thank you @fourpastmidnight for all the details here. I am in the same situation and am trying to figure out how to correct the self-signed certificate chain issue.

You mentioned this in your last post:

Older Azure DevOps Build Agents come bundled with a version of Node.JS which does not use the Windows Certificate Store. In my particular case, I do not own the build servers and therefore I am not able to update the build agent.

What is the "older" version of the build agent you are referencing?

Also, I am curious if you've ever been able to use the JFrog CLI directly on the build server. Even when I try to run something like, jf config add, from the build server I get a certificate error. I am assuming no, as you don't have control over them.

What worked for me, and this includes on the server running jf, and the pipeline, is to add the certificate bundles to $home/.jfrog/security/certs. This is a Windows '19 server so: c:\users\<usrname>\.jfrog\security\certs.

I had tried this location in the past and it didn't work. What was different this time is having the CA certificate in ca-certs.crt and intermediate CA in custom-server.crt. I wasn't aware of the custom-server.cert file prior to reading this thread.

fourpastmidnight commented 4 months ago

Hey Brett,

I got this message, but I'm not at my computer right now. But, yes, I have the answer to both of your questions. When I'm back in front of my computer, I'll post another reply with the answers to your questions.

Get Outlook for Androidhttps://aka.ms/AAb9ysg

From: Brett Slaski @.> Sent: Monday, July 8, 2024 2:04:23 PM To: jfrog/jfrog-azure-devops-extension @.> Cc: Craig E. Shea @.>; Mention @.> Subject: Re: [jfrog/jfrog-azure-devops-extension] Use Windows Certificate Store or add command line options for Server CA certs and allow tasks to accept command line arguments for the JF CLI (Issue #458)

Thank you @fourpastmidnighthttps://github.com/fourpastmidnight for all the details here. I am in the same situation and am trying to figure out how to correct the self-signed certificate chain issue.

You mentioned this in your last post:

Older Azure DevOps Build Agents come bundled with a version of Node.JS which does not use the Windows Certificate Store. In my particular case, I do not own the build servers and therefore I am not able to update the build agent.

What is the "older" version of build agent you are referencing?

Also, I am curious if you've ever been able to use the JFrog CLI directly on the build server. I am assuming no, as you don't have control over them.

— Reply to this email directly, view it on GitHubhttps://github.com/jfrog/jfrog-azure-devops-extension/issues/458#issuecomment-2214848568, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABUTP2IQX67GBXRBWT6PPA3ZLLICPAVCNFSM6AAAAABJOZRVMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJUHA2DQNJWHA. You are receiving this because you were mentioned.Message ID: @.***>

fourpastmidnight commented 4 months ago

@brettski

We are using Azure DevOps Bulid Agent v253.1. We also have v289.x. None of these come bundled with a version of NodeJS that can make use of the Windows Certificate Store.

I have been able to directly use the JFrog CLI directly on the build server. Though, after working through many issues, I find I don't actually need to use the CLI directly in our Azure DevOps build pipelines.

So, there are two issues at play here. When using the JFrog CLI, it expects the certificate to be located at ~/jfrog/security/certs. Now, on Azure DevOps build agents, this could be %USERPROFILE%/security/certs if you install the CLI manually. But, most likely, if you use the tooling bundled with the Marketplace extension, this home is $(Agent.ToolsDirectory)/_jfrog/security/certs.

If you use the JFrog CLI task in your pipeline, your self-signed CA (trust anchor/root) certificate should be located in $(Agent.ToolsDirectory/_jfrog/security/certs.

If you're using one of the other tasks, like JFrog Dotnet, using the restore or push command, then NodeJS will be used to execute the task, and then you need to us the solution I posted above to ensure that the task will have access to the required CA certificate. Even though the other steps make use of the JFrog CLI, the CLI doesn't seem to use the configured certificate located at $(Agent.ToolsDirectory)\_jfrog\security\certs. Weird. In any event, I now have everything working.

I hope that helps clarify what's going on.

jfrog / jfrog-azure-devops-extension