gregtwallace / apc-p15-tool

APC P15 Tool is an open source replacement for APC's NMC Security Wizard. It also adds additional functionality for installing certificates on APC NMC2 & NMC3.
GNU General Public License v3.0
21 stars 5 forks source link

NMC3 Support [Need Testers] #3

Closed Salvora closed 4 months ago

Salvora commented 6 months ago

Hello, I tried this for my NMC3 v3.0.0.12 card

PS D:\Downloads\apc-p15-tool_windows_amd64> ./apc-p15-tool install --keyfile my.org.key --certfile my.org.fullchain --apchost 192.168.1.120:22 --username apc --password 12345678 --fingerprint xxxx --debug
apc-p15-tool v0.4.2
install: making apc p15 file from pem
install: successfully loaded pem files
install: apc p15 file data succesfully generated
debug: ssh: remote server key fingerprint (b64): redacted
debug: ssh: remote server key fingerprint (hex): redacted
install: apc p15 file installed on 192.168.1.120:22
install: done
apc-p15-tool done

But even after rebooting the NMC interface it does not recognize the new cert.

gregtwallace commented 6 months ago

Have you tried making the file and then uploading it manually? Does the NMC use the same wizard the nmc2 does?

Salvora commented 6 months ago

Yes, I did try uploading manually but it did not work. And according to CLI docs https://www.se.com/us/en/download/document/SPD_CCON-AYCELJ_EN/

image it should work

gregtwallace commented 6 months ago

Looking at that command and the others it sounds like the nmc3 preference is to upload the p15 key and then the cert separately. The tool combines both into one file.

Have you successfully used the security wizard tool to make a p15 key+cert file and upload it?

If you do in fact need to upload the p15 key and the cert separately I’d need to modify the tool to support that.

Please also ssh into the device and cd to the ‘ssl/‘ path and tell me the file names that are in there.

Salvora commented 6 months ago

Have you successfully used the security wizard tool to make a p15 key+cert file and upload it?

I tried with Security Wizard 1.04. For private key, it is asking for a p15 key which I don't have so I am stuck there. I have a wildcard certificate. Not sure if it will work for wildcard certs.

image image

My cert files image

Please also ssh into the device and cd to the ‘ssl/‘ path and tell me the file names that are in there.

image

gregtwallace commented 6 months ago

I assume you’re using a self signed cert generated by the card. Can you send me that defaultcert.p15 file? I can compare against what the tool is doing.

Also try uploading the tool p15 via the web ui.

Also fyi the gui security wizard you posted is pretty old I think. You might want to try the CLI version. If you can’t get it working with the official tool I’m not sure there will be much I can do. You might need to post on the vendor support forum.

Salvora commented 6 months ago

I tried to use my trusted LE issued wildcard certs with the tool. So no self signed certs. I cannot copy or read the "defaultcert.p15" file even with "apc" super user.

gregtwallace commented 6 months ago

Are you using any extra Certificate Extensions (such as Must Staple)? You might also want to try without the wildcard (i.e. set the subject name / common name to your fqdn without wildcard and don't specify any other alt names).

At the end of the day, if you're unable to produce a working file with the official tools, I'm just going to be guessing what the problem is. Your best bet is to take the problem to the official Schneider Electric forums and ask for help with the official tools first. https://community.se.com/

If you get that working come back and I can compare the working file against what my tool generates and can find the difference(s).

Salvora commented 6 months ago

I see, thanks for the information. Let me see, what is going to come out of official channel.

gregtwallace commented 6 months ago

Good luck!

Aleksey-Maksimov commented 5 months ago

Hello

I confirm the problem. Uploading the p15 file via SSH to the NMC3 is successful, but the NMC3 appears to be ignoring the file. In NMC3 we need to download the certificate and private key file separately.

изображение

It would be nice if your utility had the ability to download such a thing. Thank you.

gregtwallace commented 5 months ago

Are you able to create a private key and cert file that work in your NMC3? I would need them to reverse engineer the files. This involves uploading a private key here, so certainly don't keep/trust the files on your actual device, but I do need to know the files actually work, for sure, on your device.

Additionally, please confirm if you need to restart the network interface after uploading the files, for them to actually go into use. If you do, please upload the two files to the NMC, but BEFORE you restart the interface, login via SSH and run cd ssl and then dir and let me know all of the file names returned. For example:

apc@apc>dir
E000: Success
           0 Mar 25  2022  ./
           0 Mar 25  2022  ../
        2544 May 13  3:09  defaultcert.p15

I am guessing that perhaps the UPS firmware converts the two staged files into the defaultcert.p15 file on the device.

If you can do these things and are willing to test, I can try to add support.

Aleksey-Maksimov commented 5 months ago

Are you able to create a private key and cert file that work in your NMC3? I would need them to reverse engineer the files.

I think there is nothing magical in my key and certificate files that can be examined.

A few of my observations.

Using OpenSSL, I generated a regular private key (openssl.exe genrsa -out file.key 2048) and created a request to the local certificate authority (openssl.exe req -config my.cfg -new -key file.key.pem -out file. req). Then, based on the request file file.req, I received a certificate from the CA in Base-64 format. So I have 2 files in PEM format - file.key.pem and file.cert.pem

The good news is that using your utility I was able to create a p15 file (apc-p15-tool.exe create --keyfile file.key.pem --certfile file.cert.pem --outfile file.p15) and this the file is successfully received via the NMC3 web interface. NMC3 works with this p15 file.

The bad news is that I can't see where the downloaded p15 file ends up and what happens to it on the NMC3 storage

The general structure of NMC3 storage looks like this:

apc>dir
E000: Success
        1024 May 12  2023  apc_hw21_aos_2.5.0.8.bin
     6795900 May 12  2023  apc_hw21_su_2.5.0.6.bin
       45000 Jun  2 13:12  config.ini
           0 Oct 18  2021  db/
           0 Oct 18  2021  ssl/
           0 Oct 18  2021  ssh/
           0 Oct 18  2021  logs/
           0 Oct 18  2021  sec/
           0 Oct 18  2021  fwl/
           0 Oct 18  2021  email/
           0 Oct 18  2021  eapol/
           0 Oct 18  2021  tmp/
           0 Oct 18  2021  upsfw/
           0 Aug  1  2023  certs/

The ssl directory is empty:

apc>cd ssl
E000: Success
apc>dir
E000: Success
           0 Oct 18  2021  ./
           0 Oct 18  2021  ../

The certs directory and its subdirectories are empty:

apc>cd certs
E000: Success
apc>dir
E000: Success
           0 Aug  1  2023  ./
           0 Aug  1  2023  ../
           0 Aug  1  2023  ca/
           0 Aug  1  2023  private/
apc>cd private
E000: Success
apc>dir
E000: Success
           0 Aug  1  2023  ./
           0 Aug  1  2023  ../
apc>cd ../ca
E000: Success
apc>dir
E000: Success
           0 Aug  1  2023  ./
           0 Aug  1  2023  ../

But the installed certificate works. After rebooting the NMC, nothing changes. These directories remain empty.

The sec directory is also empty. The only place where there is something similar to certificates is the db directory, there is some file keyset.p15

apc>cd db
E000: Success
apc>dir
E000: Success
           0 Oct 18  2021  ./
           0 Oct 18  2021  ../
       70656 May  9 10:29  eventcfg2.db
          30 Aug  1  2023  appid.dat
        2048 Aug  1  2023  slcache.db
        6112 Aug  1  2023  net4.db
         104 Jun  2 12:25  hw.dat
        6112 Aug  1  2023  fwver.db
       53248 Jun  2 12:24  eventdef2.db
        4081 Jun  2 12:19  keyset.p15
       10208 Jun  2 13:24  cfg4.db
        5088 Jun  2 12:17  trkeys.db
        5088 Jun  2 13:15  user2.db
gregtwallace commented 5 months ago

They must have changed something in the firmware since the defaultcert.p15 file doesn't even exist on your device, but it has an SSL cert. Are you able to download the keyset.p15 file? You can use the Putty SCP executable to try this: ./pscp user@ups.example.com:/db/keyset.p15 ./keyset.p15. I'm guessing it won't work because in other firmware I've tested they don't allow download of the sensitive .p15 files but it is worth a shot.

If you can't download that file and provide it I am at a loss. You could try uploading the .p15 file from my tool as keyset.p15, but I can't advise that. The file size of the keyset.p15 file is significantly larger than the defaultcert.p15 so there must be something else going on in that file. For what it is worth, my NMC2 doesn't have that file, so I'm guessing part of the file is the thing we're interested in + some amount of "other" stuff.

Finally, for NMC3 you might not even need my tool. See: https://community.se.com/t5/APC-UPS-Data-Center-Enterprise/SSL-Cert-for-NMC3-Web-Interface/td-p/459844 which seems to suggest you can upload certs using the native tooling.

I do see the instructions mentioning uploading a key in the .p15 format from the Security Wizard Tool. I can actually add the functionality to generate that file from a .pem key. I considered doing it originally, but it seemed unnecessary. I will work on that sometime soon, if it is helpful. Let me know.

Aleksey-Maksimov commented 5 months ago

Yes, I tried downloading this keyset.p15 file. And of course, without success :)

If you look at the document "Network Management Card 3 - CLI (Command Line Interface) Guide" https://www.se.com/us/en/download/document/SPD_CCON-AYCELJ_EN/ then there we will see the commands ssl key, ssl csr, ssl cert. We are interested in the commands ssl key -i and ssl cert - i, with the help of which, I think, we can try to upload our own certificate and key in p15 format to the NMC3.

I have several NMC3 boards in work and we could try to expand the capabilities of the apc-p15-tool utility

gregtwallace commented 5 months ago

I have added the additional key.p15 output file when using the create command and adding the --keyp15 flag. Since I have no idea how the NMC3 processes the files, I can't make the install function work. You'll have to use the native SSH commands to install the created files, but it looks like it should be pretty reasonable to script.

Please try this build and let me know.

v0.5.0-preview.zip

Aleksey-Maksimov commented 5 months ago

Thank you.

After several unsuccessful attempts, I found a working scheme. So, the sequence of actions will be like this:

Convert the PEM (Base-64) private key to P15 format with specific APC headers:

> apc-p15-tool-amd64.exe create --keyfile D:\Certs\custom.key" \
--certfile "D:\Certs\custom.cer" --keyp15 --outkeyfile "D:\Certs\custom.key.p15"

apc-p15-tool v0.5.0
create: making apc p15 file from pem
create: successfully loaded pem files
create: apc p15 file data succesfully generated
create: apc p15 key+cert file apctool.p15 written to disk
create: apc p15 key file D:\Certs\custom.key.p15 written to disk
create: done
apc-p15-tool done

The command is executed successfully. A key file in a specific P15 format is created. Please note here that an extra file "apctool.p15" is created. This is a logical error. In our case, such a file should not exist.

Next, using SCP we send files via SSH to NMC3. Important note: when transferring a key file, we cannot use a name that contains p15 (for example .p15 or .p15.key). I noticed that NMC3 immediately deletes such a file. Therefore we will change the name of the final file.

> scp "D:\Certs\custom.key.p15" apc@nmc3.host.fqdn:/ssl/nmc.key
> scp "D:\Certs\custom.cer" apc@nmc3.host.fqdn:/ssl/nmc.crt

Now we connect via SSH to NMC3 and check the files. We see that the files are in place:

apc>cd ssl
E000: Success
apc>dir
E000: Success
           0 Jul 15  2021  ./
           0 Jul 15  2021  ../
        2016 Jun  5  8:22  nmc.crt
        1420 Jun  5  8:25  nmc.key

In the current ssl directory, install the key and then the certificate:

apc>ssl key -i nmc.key
E000: Success
apc>ssl cert -i nmc.crt
E000: Success

Now we check that the certificate is installed and used as the current one:

apc>ssl cert -s
E000: Success

Certificate
-----------
    Serial Number: 4a000....
    Issuer: CN=Corp-Sub-CA1
    Validity:
        Not Before: Wed Jun  5 04:22:57 2024 UTC
        Not After : Sat Jun  5 04:32:57 2025 UTC
    Subject: CN=nmc3.host.fqdn, ...
    Subject Public Key Info:
        Public Key Algorithm: RSA (2048 bit)
        Modulus:
            bc:22:64:b8:...
            ... 2:29
        Exponent: 65537 (0x10001)
    Subject Alternative Name:
        DNS: nmc3.host.fqdn
        DNS: nmc3
        IP Address: 10.10.5.2
    Thumbprint: 1d461f...
    Fingerprint: ed98cee...
apc>exit

There is no need to reboot the NMC3. The certificate starts working immediately.

gregtwallace commented 4 months ago

Thanks, this is very helpful. I can probably actually update the install function using this information.

Also, I'd left the certificate.p15 generating because I thought you'd need both files. I might just remove the key flag and always make it create both files. I don't know. We'll see :)

gregtwallace commented 4 months ago

Give this one a shot. I think install should work now. Let me know how it goes.

v0.5.0-preview2.zip

gregtwallace commented 4 months ago

Were you able to try preview 2?

Aleksey-Maksimov commented 4 months ago

Hello.

Does not work. The process starts but does not end.

C:\Temp\v0.5.0-preview2>apc-p15-tool-amd64.exe install --keyfile "d:\certs\up006.holding.com.key" --certfile "d:\certs\up006.holding.com.cer" --apchost "up006.holding.com:22" --username "apc" --password "MyPW!0rd" --debug --fingerprint "7Z3baEepWWbGRZjq7iSshMEuUHzghnFx4HBVNdNX69g"

apc-p15-tool v0.5.0
install: making apc p15 file from pem
install: successfully loaded pem files
install: apc p15 file data succesfully generated

At this point the process hangs

gregtwallace commented 4 months ago

I adjusted a little bit more (and found a typo in my regex). Please try attached preview 3.

If this doesn't work -- please SSH into your UPS (using MobaXterm, Putty, or whatever client you prefer) and then type ssl and push enter.

Then copy everything in the terminal up to that point and provide it here. I'm guessing something that works on my NMC2 isn't quite the same on NMC3. Please do not redact anything (there isn't anything sensitive) as I need to know the exact content. If you absolutely must change something please make it clear exactly what changed and how. For example, substitute example.com for your FQDN.

Here is an example of what I'm asking for (from my NMC2):

Schneider Electric                      Network Management Card AOS      v7.1.2
(c) Copyright 2023 All Rights Reserved  Smart-UPS & Matrix-UPS APP       v7.1.2
-------------------------------------------------------------------------------
Name      : apc9138A8                                 Date : 06/18/2024
Contact   : Unknown                                   Time : 10:32:26
Location  : Unknown                                   User : Super User
Up Time   : 11 Days 21 Hours 52 Minutes               Stat : P+ N4+ N6+ A+
-------------------------------------------------------------------------------
IPv4               : Enabled            IPv6               : Enabled
Ping Response      : Enabled
-------------------------------------------------------------------------------
HTTP               : Disabled           HTTPS              : Enabled
FTP                : Disabled           Telnet             : Disabled
SSH/SCP            : Enabled            SNMPv1             : Disabled
SNMPv3             : Disabled
-------------------------------------------------------------------------------
Super User         : Enabled            RADIUS             : Disabled
Administrator      : Disabled           Device User        : Disabled
Read-Only User     : Disabled           Network-Only User  : Disabled

Type ? for command listing
Use tcpip command for IP address(-i), subnet(-s), and gateway(-g)

apc@apc>ssl
E101: Command Not Found

apc@apc>

v0.5.0-preview3.zip

Aleksey-Maksimov commented 4 months ago

It worked.

apc-p15-tool v0.5.0
install: making apc p15 file from pem
install: successfully loaded pem files
install: apc p15 file data succesfully generated
install: connected to ups ssh, installing ssl key and cert...
install: apc p15 file installed on up006.holding.com:22
apc-p15-tool done

On the NMC3 web page, after F5, the new certificate is used.

gregtwallace commented 4 months ago

@Aleksey-Maksimov awesome! Thanks for confirming. I will include this in the next full release, likely in the coming days.

@Salvora can you test v0.5.0-preview3.zip with your NMC3 ?

Salvora commented 4 months ago

@Aleksey-Maksimov awesome! Thanks for confirming. I will include this in the next full release, likely in the coming days.

@Salvora can you test v0.5.0-preview3.zip with your NMC3 ?

Currently, I am away from home and wil lbe back in August. I can test this only when I get back. I will get back to you regarding the results.