search

vapor-ware / synse-snmp-plugin

SNMP plugin for Synse
GNU General Public License v3.0
1 stars 2 forks source link

v3: re-enable CI test stage #40

Closed edaniszewski closed 4 years ago

edaniszewski commented 4 years ago

the test stage in CI has been disabled temporarily, as the CI pipeline in the v3 branch was updated to use the newer jenkins setup where agents are run in k8s. That change doesn't allow the same docker build/run/networking as was doable before (which the tests relied on), so we will need to find a different approach that satisfies the needs of the test env. Doesn't seem too difficult, just needs some thinking through. Tackling this later, as there are some other higher priority things I'd like to get done before spending time on this.

MatthewHink commented 4 years ago

@edaniszewski @marcoceppi @lazypower What can we do to rectify this? Now we have CI that doesn't run tests.

edaniszewski commented 4 years ago

I think this is a related issue: https://github.com/vapor-ware/cloud-ops/issues/143

lazypower commented 4 years ago

I found the blocking issue - and it is related to the image being minted in the synse-snmp-plugin project vs what we publish for a synse-snmp-emulator image.

I'm going to campaign that we stop doing "one off" emulators in projects like this, and instead focus on keeping the project for the synse-snmp-emulator up to date. It contributed to a loss of 4 manhours debugging the issue that only occurred in CI because we were trying to use the vaporio/synse-snmp-emulator:latest image, which apparently doesn't pass the test suite as the one minted in the repository.

What i did to correct/overcome this:

Then consumed that image in the .jenkins file for CI and observed passing results. :disappointed:

Messaging received from the vaporio/synse-snmp-emulator image:

Response var-binds: 1.3.6.1.2.1.33.3.2.3=No Such Instance currently exists at this OID

MatthewHink commented 4 years ago

We're going to disagree here. I am going to take the position that CI needs to run tests that validate the product.

We have many many tests that use emulators. Some are disabled right now due to agent any. Some are still running with agent any.

We often need to test the protocol level. Failing real hardware to test against, we use emulators. There are going to be a lot more than 4 man hours fixing bugs once the tests get reactivated.

It's just not right that CI can't run tests against emulators when it did so in the past. Furthermore if CI cannot run the tests, how do we validate functionality? In that case, CI is out, which seems wrong.

If the problem is that agent any requires a full VM rather than containers (and it may not be), can we just add hardware for it?

lazypower commented 4 years ago

I think i struck a nerve and that wasn't my intent. I'm sorry, and I'll own that I wasn't doing a very good job of making my intent clear:

What I was saying above is that this specific issue, was related to the difference in the embedded emulator in this project (successfully validates the test suite), vs the snmp-emulator repository that was provided for runtime (does not, is missing a MIB or something).

So disagree away :) I'm not telling you to not use emulators, disable tests, or only conform to K8s. What i'm attempting to do is make our CI process scale in a repeatable fashion with a reliable subscription stream (semver usage of ci-shared) and in order to make things conform to that process, making it run in k8s is and should be our first stop. We can investigate alternatives when it doesn't conform to the rest of our build processes.