State of local MIB -> JSON support?

[remingtonc]

Hi, wondering about the state of JSON support. Noticed it's not in v0.0.7, so pulling off of master branch. Trying to generate JSON files from local MIBs but am not having success. The index gets generated but there's no content in it. Using the following code:

def transform_mibs_to_json(self, specific_mibs=None, exclude_mibs=['test-mib.my']):
    mib_names = []
    if specific_mibs is None:
        for filename in os.listdir(self.local_mib_dir):
            try: filename.rindex('.my', -3)
            except: continue
            mib_names.append(filename)
    else:
        mib_names = specific_mibs.copy()
    try:
        mib_compiler = MibCompiler(
            SmiV1CompatParser(),
            JsonCodeGen(),
            FileWriter(self.local_json_dir).setOptions(suffix='.json')
        )
        mib_compiler.addSources(FileReader(self.local_mib_dir, recursive=True))
        mib_stubs = JsonCodeGen.baseMibs
        searchers = [AnyFileSearcher(self.local_json_dir).setOptions(exts=['.json']), StubSearcher(*mib_stubs)]
        mib_compiler.addSearchers(*searchers)
        if not os.path.isdir(self.local_json_dir):
            os.makedirs(self.local_json_dir)
        processed = mib_compiler.compile(
            *mib_names,
            **dict(
                noDeps=False,
                rebuild=True,
                genTexts=True,
                writeMibs=True,
                ignoreErrors=False
            )
        )
        mib_compiler.buildIndex(
            processed,
            ignoreErrors=False
        )
    except error.PySmiError:
        sys.stderr.write('ERROR: %s\n' % sys.exc_info()[1])
        sys.exit(-1)

See anything wrong in usage? Will try and submit a fix if it's a bug. Any pointers or thoughts?

[etingof]

Quick question while I'm looking into your code: have you tried the mibdump.py tool for JSON generation? I'm running it for both MIB and index and it works for me...

[etingof]

My guess is that pysmi compiler can't find dependent MIBs. To figure that out you should look into the processed dictionary -- it carries status and indicates missing dependencies.

My other suggestion would be to enable debugging:

from pysmi import debug
debug.setLogger(debug.Debug('reader', 'searcher', 'compiler'))

so it will show you (on stderr) which MIBs it wants to load, where it tries to look them up and ultimately failed.

Once you give it access to all MIBs your MIB depends on, it should be able to compile everything into JSON. Let me know if you need help with this.

[remingtonc]

@etingof Excellent advice, the dependencies were indeed the issue. Enabling debugging is also extremely useful.

I am running in to some issues building the MIBs - 282/1147 that I am compiling are failing. Ignore the mega grep...

(python3)(python3)cat output.txt | grep "error from" | sed "s/\>\: /\\$(echo -e '\n\r')/" | sed "s/ at MIB/\\$(echo -e '\n\r')/" | grep "no" | sort -u | uniq -u
no module "AWCVX-MIB" in symbolTable
no module "TOKEN-RING-RMON-MIB" in symbolTable
no symbol "Gauge32" in module "SNMPv2-SMI"
no symbol "IpAddress" in module "SNMPv2-SMI"
no symbol "TimeTicks" in module "SNMPv2-SMI"
no symbol "Unsigned32" in module "SNMPv2-SMI"
no symbol "ciscoAgentCapability" in module "CISCOWAN-SMI"
no symbol "zeroDotZero" in module "SNMPv2-SMI"

Should I ignore these errors, or will this present serious compilation issues?

Clarification: Running with ignoreErrors as True results in compilation working. Will these errors actually effect the compilation? Or is passing ignoreErrors just saying to do a shallow run and not verify the actual consistency of the data and just parse what the MIB contains?

[etingof]

You should not be getting errors for SNMPv2-SMI, not so sure about CISCOWAN-SMI (will look that up).

The situation is this: MIBs are built on top of SNMP-specific data structures (like OBJECT-TYPE) that are described in SNMPv2-SMI in a semi-formal language. Implementers are supposed to express those data structures in their mother tongue and ship them as part of SNMP/MIB library. Therefore there is no much sense for MIB compiler to process SNMPv2-SMI (and a few other MIBs of the same kind). The approach we have with pysmi is to pretend that those MIBs are already compiled so the compilation is always waved (this is done with so-called StubSearcher).

This makes me wonder why it seems like pysmi does not fake those base SNMP types that SNMPv2-SMI supplies... It may be a bug in pysmi, though I'm not getting it with mibdump.py.

It's harder to debug this without having a reproducer. Could you please add one to this issue? I will play with it then. ;-)

[remingtonc]

@etingof This gist should reproduce the experience, the only difference being that the ignoreErrors flag should be changed to False for the errors to be output (didn't build in that capability).

The situation is that I am downloading all of the Cisco MIBs from the public FTP server and compiling them. I can supply you with the compiled JSON MIBs for the JSON folder on the pre-compiled directory of the web server if you would like to save your own processing time. Thanks for this great library! If this is indeed a bug I will start a separate issue and we can close this out.

[remingtonc]

Closing as conversion of MIBs to JSON is supported and continuing issue is tangential.