SNMPv3 USM support

[jonnystorm]

The biggest hurdle right now is USM engineID discovery, which effectively does not exist in OTP.

In Erlang/OTP master, :snmpm.discovery is commented out, with calls pointing to :snmpm_server which contains no implementation. It isn't yet clear to me whether code in :snmpa.discovery could be useful, but that's likely the closest thing OTP has.

[jonnystorm]

Where I'm stuck today: https://gist.github.com/jonnystorm/aa39a241a375975ce78b2a390fdf991f.

[alexnavis]

Will have a look at this and update you.

[alexnavis]

I will start working on this, if you are not already working on this.

[jonnystorm]

By all means, have at. :) I've just focused on basic housekeeping, recently.

Would you care to become a collaborator? I can get you added if you're interested.

[alexnavis]

Yes, please I wanted to ask you as well. I thought I will contribute something before asking :).

Thanks, Alex

[jonnystorm]

Done. :) Thank you!

[alexnavis]

Thank you. :)

[alexnavis]

Hi Jonny,

I spend some time to understand the USM and to make it work. I'm currently stuck at a problem, but I list out my findings,

I tried with a local switch here, I have summarized the problem with web snmp simulator. I have Below IP address and engine ID belongs to snmp simulator from this URL, so it will work for you (http://snmpsim.sourceforge.net/public-snmp-simulator.html)

1) For v2, I tried the following and it worked. (engine id doesn't matter, it is dropped in UDP).

:snmpm.register_agent('default_user', 'default_agent', [engine_id: '80004fb805636c6f75644dab22cc', address: [104,236,166,95], community: 'public', version:  :v2, sec_model: :v2c])
:snmpm.sync_get('default_user', 'default_agent', [[1,3,6,1,2,1,1,1,0]])
:snmpm.sync_get('default_user', 'default_agent', [[1,3,6,1,2,1,1,1,0]])

2) For v3, I tried the same with a similar configuration which timed-out.

:snmpm.register_user('default_user', :snmpm_user_default, nil, [])
engine_id = '80004fb805636c6f75644dab22cc'
user_name = 'usr-md5-none'
password  = 'authkey1'
auth_key = :snmp.passwd2localized_key(:md5, password, engine_id)
:snmpm.verbosity(:all, :trace)
:snmpm.register_usm_user(engine_id, user_name, [auth: :usmHMACMD5AuthProtocol, auth_key: auth_key, priv: :usmNoPrivProtocol, priv_key: []])

:snmpm.register_agent('default_user', 'default_agent', [engine_id: engine_id, address: [104,236,166,95], version:  :v3, sec_model: :usm, sec_name: user_name, sec_level: :authNoPriv ])
:snmpm.sync_get('default_user', 'default_agent', [[1,3,6,1,2,1,1,1,0]])

while a SNMP tool works for the same,

snmpget -v 3 -u "usr-md5-none" -e 80004fb805636c6f75644dab22cc -a MD5 -A "authkey1" -l authNoPriv 104.236.166.95 1.3.6.1.2.1.1.1.0

I compared the UDP packets sent via both using wireshark. Somehow the packets header 'authoritativeHeaderID' which is created from the engineID is encoded differently in the evm which is the reason for the failure.

I'm still trying to going through the erlang code to understand it. Just wanted to update on you this. Also SNMP discovery is only available for the SNMP manager module (available in the agent module).

Thanks, Alex

[jonnystorm]

Does authoritativeHeaderID appear in the Erlang source, or did it appear in Wireshark? If this is somehow related to the authoritative engine ID, then see the textual convention for SnmpEngineID.

What I take from RFC 3411 is: no particular algorithm is prescribed, and each enterprise may strive for uniqueness however they choose. This makes engine ID discovery paramount for USM, as we cannot predict how a particular device will make its engine ID "authoritative."

I think we should document these kinds of problems as tests, which means we need to achieve parity with the all the options recognized by various :snmpm functions. If you're not already working in that direction, I'll devote some time tomorrow to making this kind of exploration easier. In essence, I want us to have good tools for understanding these interactions, and those tools should be part of snmp-elixir.

Right, since the :snmpm.discovery/x functions aren't really implemented, we may be able to lean on what was done in :snmpa, but the gist I initially provided documents where that fails, and I have yet to dig deeper into what I was missing. Ultimately, we may still need to wrest control of the socket from OTP, but I'd rather avoid it.

As ever, thanks for taking taking the time to investigate this!

[alexnavis]

My bad, it is 'msgAuthoritativeEngineID' and I have attached snapshot for working one via snmpget and non-working one(yellow) based on the above code example.

I wanted to go with the explicit engineID approach to see if it working and helps for my understanding. I can see the discovery been done by the 'snmpget' tool when I don't mention the engine-id, it makes an first call to discover the engineId.

Let me add some them as tests for the same for what ever I have done. Good idea.

snmpm.discovery - yeah I get the idea. I will look in to it once I resolve the above issue.

Thanks for the help 👍 .

[jonnystorm]

19 should remove some of the friction with experimentation. I also did some much-needed triage around perform_snmp_op/5, which should make the code a little easier to understand.

Here is a successful noAuthNoPriv query.

Then here is a capture of an unsuccessful authNoPriv query. Notice how the SNMP server recognized the request as noauth despite the flags, authoritative engine ID, and authentication parameters being correctly set.

I suspect the erroneous engine boots and engine time values--both 0--are what will need attention next, and it isn't yet clear to me how we can access these values without handling the PDUs directly. :snmpm_config.get_usm_eboots/1 and :snmpm_config.get_usm_etime/1 may be able to provide some answers, but more research is needed.

[alexnavis]

Thanks for the update and changes Jonny. Sorry I was held-up since last week and didn't do much. Will be looking in to this root cause of this.

Thanks, Alex

[jonnystorm]

That's hardly anything to worry about, Alex: you and I are both working for free here. :)

As ever, thanks for all your help!

[alexnavis]

Hi Jonny,

Just to update you. I was able to make the v3 work with engine_id using snmpm. SNMP credentials are the same as public SNMP simulator. https://gist.github.com/alexnavis/4bd8a5e3b6c924f72ad51036f5e990ac

Also have found a way to do the discovery with snmpa module to discover the engine id and link the above together. It needs agent.conf and standard.conf for now. I will create a PR for the same.

One caveat: This works for MD5, SHA and DES. But AES has a pending bug in erlang, because of that it doesn't work. Refer this - http://erlang.org/pipermail/erlang-patches/2014-June/004683.html

Thanks, Alex

[jonnystorm]

Ah, that makes sense. I'm glad to see I was wrong, and the time value is a non-issue.

I look forward to seeing your solution. I never was able to cut the wheat from the chaff in :snmpa. :/

Are you certain that bug is causing us trouble? They seem to indicate the current behavior is correct for GET and SET:

The agent is currently always using the local engine to get engine boots and engine time, which happens to be correct for GET, SET, and TRAP, but is wrong for INFORM.

I'll try to catch up tomorrow and better understand where things break down with AES.

Thank you for the update, Alex. With your help, I feel like a sensible Elixir SNMP client is finally within reach!

[alexnavis]

Finding the engine id decoding was a bit tricky (wireshark was the life saver :)).

Thanks for pointing that. My GET didn't work only for AES, I tried different combinations, I found that AES salt was using local_engine_id. So that lead to my wrong conclusion. I think I misread the comments, I will re-verify this again.

Happy to help and see this library coming through. Thanks man..

[alexnavis]

Couldn't make the AES work. Struggling with this.

From the erlang code for AES, it uses Local EngineBoots and EngineTime to create the IV. SaltFun() is a incremental value which is sent as part of the authorizationParameters in the UDP headers.

snmp_usm.erl.
aes_encrypt(PrivKey, Data, SaltFun, EngineBoots, EngineTime) ->
    AesKey = PrivKey,
    Salt = SaltFun(),
    IV = list_to_binary([?i32(EngineBoots), ?i32(EngineTime) | Salt]),
    EncData = crypto:block_encrypt(?BLOCK_CIPHER_AES, 
                   AesKey, IV, Data),
    {ok, binary_to_list(EncData), Salt}.

https://gist.github.com/alexnavis/8eec113cabc47a43a5a6d1eb870352fb

[alexnavis]

https://www.ietf.org/rfc/rfc3826.txt

3.1.2.1. AES Encryption Key and IV

The first 128 bits of the localized key Kul are used as the AES encryption key. The 128-bit IV is obtained as the concatenation of the authoritative SNMP engine's 32-bit snmpEngineBoots, the SNMP engine's 32-bit snmpEngineTime, and a local 64-bit integer. The 64- bit integer is initialized to a pseudo-random value at boot time.

The IV is concatenated as follows: the 32-bit snmpEngineBoots is converted to the first 4 octets (Most Significant Byte first), the 32-bit snmpEngineTime is converted to the subsequent 4 octets (Most Significant Byte first), and the 64-bit integer is then converted to the last 8 octets (Most Significant Byte first). The 64-bit integer is then put into the msgPrivacyParameters field encoded as an OCTET STRING of length 8 octets. The integer is then modified for the subsequent message. We recommend that it is incremented by one until it reaches its maximum value, at which time it is wrapped.

An implementation can use any method to vary the value of the local 64-bit integer, providing the chosen method never generates a duplicate IV for the same key.

A duplicated IV can result in the very unlikely event that multiple managers, communicating with a single authoritative engine, both accidentally select the same 64-bit integer within a second. The probability of such an event is very low, and does not significantly affect the robustness of the mechanisms proposed.

[alexnavis]

Found the fix for the above problem. I tested locally as well, AES works well now. Needs a patch in erlang.

https://github.com/alexnavis/otp/tree/fix_snmp_v3_aes (last 2 commits)

Full thread on similar issue - https://groups.google.com/forum/#!searchin/erlang-programming/SNMP$20v3$20usmStatsNotInTimeWindows%7Csort:relevance/erlang-programming/8n3CYjwi1aE/aYEe-CWdCAAJ

[jonnystorm]

This is fine detective work, Alex. Thank you so much!

I just merged #24, which cleans up USM credential handling and adds integrated tests for all security levels. Once we incorporate USM engine discovery, we can modify the tests to remove the static engine IDs. For now, the tests will act as a guidepost for further work.

If there's anything I can do to help with the Erlang contribution process, please let me know!

[alexnavis]

Thanks @jonnystorm . No more thanks :). Haven't done as much contribution as you have helped us to start things. Appreciate it.

Have created a PR for the v3 and have a look. Please do give your feedback. Have couple of items to discuss as well.

For the erlang PR, i might need help. I have created the final commits and trying to run the tests. There are some test failures. Will share more on that.

Thanks, Alex

[jonnystorm]

@alexnavis Merged #26! :D

[jonnystorm]

It's been awhile, but in revisiting this, I was able to test this patch, posted in 2016, against the latest OTP maint branch; it worked flawlessly.

Consequently, I've reinstated AES in integrated tests, which now pass with a patched Erlang. Soon, I'll submit a PR to OTP, and if all goes well, SNMP USM will be fully working after the next Erlang/Elixir release cycle. As such, I'm considering the matter closed. Finally, this can be checked off our list.

[alexnavis]

Good Jonny, that was definitely a nagging issue. Bad that I couldn't continue the erlang PR.

[jonnystorm]

Agreed! But I wasn't exactly on top of things, myself. ;)

Here it is: https://github.com/erlang/otp/pull/1874. I certainly couldn't have done it without you laying so much of the groundwork.

Thanks for stopping back by; now we both can celebrate!

[alexnavis]

Awesome to see the PR. Cheers. 🍻 👍

[Kintull]

Oh, guys, you did a great job!