sdnfv / openNetVM

A high performance container-based NFV platform from GW and UCR.
http://sdnfv.github.io/onvm/
Other
263 stars 135 forks source link

Unable to re-launch NF with same instance ID #233

Closed catherinemeadows closed 3 years ago

catherinemeadows commented 4 years ago

Bug Report

Current Behavior When re-launching an NF with the same instance ID (from either command line or from a JSON config file), the manager exits with the errors:

RING: Cannot reserve memory 
RING: Cannot reserve memory
RING: Cannot reserve memory
EAL: Error - exiting with code: 1
Cause: Cannot create rx ring queue for NF 1

and the NF hangs on

APP: Waiting for manager to assign an ID...

This bug occurs when re-launching an NF with the same instance ID (even if the service ID is changed).

Expected behavior/code NFs should successfully re-launch if assigned the same instance ID as a prior launch.

Steps to reproduce Launch NF with specified instance ID (either command line or from a JSON config file), terminal with Ctrl+C, and re-launch with the same instance ID.

Command line instructions for running speed_tester:

$ ./go.sh -- -n 1 -r 1 -- -d 2

Terminate with Ctrl + C and relaunch:

$ ./go.sh -- -n 1 -r 1 -- -d 2
Screen Shot 2020-06-16 at 11 29 22 AM

The same error occurs when re-launching an NF with the same instance ID but a different service ID:

$ ./go.sh -- -n 1 -r 1 -- -d 2

Terminate with Ctrl + C and relaunch:

$ ./go.sh -- -n 1 -r 2 -- -d 2

Environment

Possible Solution

Make sure rings are being freed on NF shutdown

Additional context/Screenshots

The screenshot shows the result of trying to re-launch speed_tester from a JSON config file with the same instance ID as the first successful launch:

$ ./go.sh -F ../example_config.json -- -- -d 1
$ Ctrl + C
$ ./go.sh -F ../example_config.json -- -- -d 1
rx_ring_bug
catherinemeadows commented 4 years ago

@bdevierno1 this might be a fix that can be included in your PR

dennisafa commented 4 years ago

Yea this ones on me - when I moved the NF ring allocation from the manager to the NF, I didn't account for the fact thats rings need to be allocated from a secondary process. In my latest PR (NF pool) i wrote a function that allocates a single ring from the manager using a NF -> manager communication mechanism. When we merge that we should fix this issue by having the NF use that function whenever allocating the ring.

kevindweb commented 3 years ago

Fixed from merge in #272