Resolve MIB object names to OIDs

[jonnystorm]

Here's the procedure I put together for yeongsheng-tan a while back regarding net-snmp-elixir.

Gotta' find a better way to import rowStatus in all those knackered MIBs. And that's hardly the only problem we'll encounter.

Keeping a net-snmp installation's worth of compiled MIBs in memory is not an option, which means we'll have to put them somewhere, which means the location should be configurable, which means we'll have to call it something. How about compiled_mib_path? Or maybe compiled_mib_dir? Something. Same goes for the path to the source MIBs.

I think sidestepping any deep thinking about keeping compiled MIBs synchronized with their sources is all right for now. The most optimization I might do is to only compile those MIBs (+ dependencies) for which we don't have compiled counterparts. That way, sync problems can be fixed by just deleting files in the compiled MIB path. I don't recall: is it even that costly? Can we just take the hit every app start? Hmm...

[alexnavis]

The usecase I can only think is for the OID to Name mapping. Are there any other use cases ?

I like the way how ruby library has done this, it has all the default standard mappings in yaml instead of parsing it every time (in our case it could be .bin file). For custom MIB's it can be parsed and loaded be on demand. Any thoughts ?

https://github.com/hallidave/ruby-snmp/tree/master/data/ruby/snmp/mibs

[jonnystorm]

I claim name->OID is most important (I made it "wontfix" in net-snmp-elixir), but enumerations for OID values is also up there. OID->name is less important to me, but Erlang gives it to us for free.

The gist I linked contains the procedure for MIB compilation, which is fully supported in Erlang. This is a one-time cost, so long as a cache of compiled MIBs is sensibly maintained.

snmp-mib-elixir was my attempt to do what ruby-snmp appears to have done, which is essentially recreate MIBs in a more digestible format. The problems I encountered with this approach are: it is error-prone when done by hand; it is error-prone when done by code generation; it is impoverished compared to what OTP has already done. Compare IP-FORWARD-MIB.yaml to what OTP has out of the box, and you begin to develop a sense of just how far the OTP maintainers went.

Take a look at :snmpc.compile/2 for the MIB compiler API, which I have barely begun to scratch the surface of, and also see :snmpm.name_to_oid/1 et al for post-compile functionality. Near as I can tell, 80-85% of what we need is there, and someone just needs to make the parts around the MIB compilation decent. To do that, we need opinionated, reliable automation to handle the dependency tree, aberrations in the base MIBs, and errors in the rest (and there are plenty of those).

Failing the above, users will be forced to finagle, exactly once, every new MIB they need, and I would rather offer them partial functionality with apologies. Doing such one-off finagling is what got me here in the first place.

[jonnystorm]

See pull request #7 for a first attempt at handling the compiling. And here are all the errors that will need to be dealt with.

It would be nice to eventually pull missing MIB files from https://github.com/trevoro/snmp-mibs or somesuch.

[alexnavis]

Thanks for the details on snmp:compile, still trying to read more on this.

With respect to error in MIB compilation, I had similar errors using 'smidump' or snmp:compile, but a colleague of mine mentioned that these errors are because of the dependency between the MIBs. If you put them in the right order it might work. I haven't tested this enough, will try out for couple of the MIBs and let you know.

[jonnystorm]

Pull request #10 adds an integrated test for compiling the base net-snmp MIBs.

Ideally, we would parse the syntax errors emitted by :snmpc.compile/2, but I haven't worked out how to capture them.

[jonnystorm]

Pull request #11 excludes integrated tests by default. The aforementioned test can now be run with mix test --include integrated.

[jonnystorm]

^^^ comment fail

[alexnavis]

Had a look at your code & snmp compiler. Did some tests to understand it better. You might already know the below.

Updates from my analysis on the mib compilation errors:

• No RowStatus column in table (is introduced in SMIv2 - looks like a warning which can be ignore)
• 'memTotalReal' missing in OBJECT-GROUP (E.g., tried to compile UCD-SNMP-MIB.mib, UCD-SNMP-MIB.txt).
  • With "verbosity: :trace", it shows '[UCD-SNMP-MIB.mib][snmpc:1480][LOG]: save failed: yes'. Also :snmpc.is_consistent([erl_mib_file]) seems to be true. Not sure?
• File not found is straightforward as the dependent bin file are not found.

[jonnystorm]

Updated gist with errors (using warnings: false): https://gist.github.com/jonnystorm/eaa560ff971b6b124eb611c8faf584ba. I'll add some mix configuration to set the MIB compiler output later.

I hope to commit proper dependency-handling soon. The algorithm I worked out this morning correctly handles cyclic dependencies in O((1/2) * n^2) time, which takes care of the most obvious class of problems.

I'll add the consistency check to the integrated test; it should at least be there.

I know I won't be able to attack any of the other errors this week; feel free to hack something together if time allows.

[jonnystorm]

Useful discussion of TEXTUAL-CONVENTION import errors.

[alexnavis]

Interested to see how you solve the dependency problem, is this a extension of the file loading based on module imports ?.

[jonnystorm]

I don't believe so, if I understand your question correctly.

The basic idea is this:

1. Obtain mappings for all imports.
2. For each MIB, check the mappings to see if it's imported by another MIB.
3. Compile all imported MIBs without imports of their own.
4. Go back to (2) with the MIBs we didn't compile.

Picture a modest dependency tree:

e

d
 \
  )=>b
 /
a
 \
  `-> c

The import mappings for this become

b -> {ad}
c -> {a}

where e is not present, as it has no imports. Now we can walk all the mappings simultaneously.

Start with the set of all MIBs.

{abcde}

Replace each MIB with any MIBs that depend on it. Remove from the set any MIBs without dependents.

{ada}

The MIBs that dropped out, {bce}, can be compiled first. Walking the mappings again gives

{}

and thus we may compile a and d without worry.

In the case of cyclic dependencies, we can detect a loop by testing for set equality. Consider a cycle.

a<----b
 \    ^
  \   |
   `->c

Here, we start with a set {abc}, but walking the mappings produces a set {bca}, which is clearly just a permutation of the original set. The moment we detect set equality between the last set and the current set, we can raise an exception.

I don't think we'll ever encounter cyclic dependencies in MIB imports (it would certainly constitute an error in the MIB, if we did), but the loop detection mechanism is simple enough, and I would rather avoid forcing users to kill processes if the worst should happen.

Does all of that make sense? If I didn't state something clearly, please let me know.

[jonnystorm]

FYI: The above algorithm is implemented as of pull request #15.

[alexnavis]

Thanks for spending time to give a detailed explanation. I understand what you are trying to do and had similar idea in mind for the dependency graph. As you said there can't be proper MIBs with loops and hence I feel the case of loops might be non-real world scenario. Will look at the implementation. Thanks.

[jonnystorm]

According to this, IPV6-TC is obsolete. This is confirmed by RFC 8096. Since our test fixture for INET-ADDRESS-MIB is up to date, I may log a warning about the obsolete MIBs with the reference RFC and return an :obsolete_mib error.

[jonnystorm]

As of #20, all MIBs we care about (for now) are compiling, and as of #21, we have a couple basic functions to resolve object names. All that remains is to make it easy for folks to set a MIB path, leaving the compiling and loading to us.

[jonnystorm]

22 makes the following possible:

iex> SNMP.start
:ok
iex> SNMP.resolve_object_name_to_oid :ipCidrRouteTable
{:ok, [1, 3, 6, 1, 2, 1, 4, 24, 4]}

• The default MIB cache (configured in the :mib_cache environment variable) is /tmp/snmp_ex/mibs.
• The default MIB source (configured in the :mib_sources environment variable) is /usr/share/snmp/mibs.

The API may need some polishing, and it may be beneficial to check directories and recompile/reload MIBs at runtime, but I think what we have now fulfills the intent of this issue. As such, I'm happy to close this up.