multi_usrp::make initializes/resets device not properly

[Kuurusch]

Issue Description

It is not possible to restart our application and initialize the USRP's after the application crashed or was terminated by the user with ctrl-c. It seems, the USRP's are not properly reset during initializing the USRP's again.

Setup Details

We are working on Ubuntu 18.04 LTS and have installed UHD 3.14 rc. We use RFNoC in combination with multi_usrp (5 USRP's x310 with daughterboard ubx-160). We tested the USRP's with the standard DDC- and a customized PSD-block (power spectral density). Samples per packed and per buffer is set to 256, because one PSD has 256 bins.

Expected Behavior

The USRP's should be able to reinitialize without powercycle. So when the application crashes or is terminated in terminal with ctrl-c, it should be possible to restart the application, without powercycle the hardware. Under UHD 3.9 and with our custome FPGA was this no problem.

Actual Behaviour

After a powercyle of the devices the initialization and the start of an rx_stream works just fine and we get the expected data. But if the application crashes or we kill it with ctrl-c in the terminal, a second restart of the application is not possible. Initialization and start of the rx_stream works just fine, but after several received packages with the recv() function the application stuck after a further call of the recv() function and it prints out repeatedly the following error:

[ERROR] [STREAMER] The receive packet handler failed to time-align packets. 1002 received packets were processed by the handler. However, a timestamp match could not be determined.

The recv() function never returns.

Because it happens only after several successfully received packages, I think it's a problem with a buffer. So a buffer is overwritten partially and the data are corrupt. Thats maybe the reason, why it cant's time-align the packages anymore. But we resized the linux socket-buffer already with: sudo sysctl net.core.rmem_max=33554432

Steps to reproduce the problem

We initialize the USRP's with the following code/steps:

//================= //Set-up streaming: //================= uhd::device_addr_t streamer_args(streamargs); std::vector rx_channel_nums;

rx_graph = usrpDevice->create_graph("rx_graph");

for(int i = 0; i < numChannels; i++)
{
    /*//Check if agc0-block is available:
    if(!usrpDevice->has_block(agc0_id))
    {
        qInfo() << endl << "AGC-block " + QString::number(i % 2) + " does not exist on device " + QString::number(qFloor(i/2)) + "!" << endl;
        throw "AGC-block " + QString::number(i % 2) + " does not exist on device " + QString::number(qFloor(i/2)) + "!";
    }

    uhd::rfnoc::source_block_ctrl_base::sptr agc_block_ctrl = usrpDevice->get_block_ctrl<uhd::rfnoc::source_block_ctrl_base>(agc0_id);
    agc_block_ctrl->set_args(uhd::device_addr_t(agc_args)); //set arguments of agc
    //hier sollten in einer for-Schleife die Noise-Floors Byte, für Byte gesetzt werden...
    rx_graph->connect(radio_ctrls[i]->get_block_id(), 0, agc_block_ctrl->get_block_id(), 0);   //connect radio with agc*/

    //Radio -> DDC:
    //=============

    //Check if ddc-block is available on current device:
    if(!usrpDevice->has_block(uhd::rfnoc::block_id_t(std::to_string(qFloor(i/2)) + "/DDC_" + std::to_string(i % 2))))
    {
        qInfo() << endl << "DDC-block " + QString::number(i % 2) + " does not exist on device " + QString::number(qFloor(i/2)) + "!" << endl;
        throw "DDC-block " + QString::number(i % 2) + " does not exist on device " + QString::number(qFloor(i/2)) + "!";
    }

    uhd::rfnoc::source_block_ctrl_base::sptr ddc_block_ctrl = usrpDevice->get_block_ctrl<uhd::rfnoc::source_block_ctrl_base>(uhd::rfnoc::block_id_t(std::to_string(qFloor(i/2)) + "/DDC_" + std::to_string(i % 2)));

    ddc_block_ctrl->set_args(uhd::device_addr_t("freq=0,input_rate=" + std::to_string(multiUSRP->get_master_clock_rate()) + ",output_rate=" + std::to_string(multiUSRP->get_master_clock_rate() / _ip->decimation))); //set arguments of ddc
    rx_graph->connect(radio_ctrls[i]->get_block_id(), 0, ddc_block_ctrl->get_block_id(), 0, 1);   //connect radio with ddc

    //DDC -> PSD:
    //===========

    //Check if psd-block is available:
    if(!usrpDevice->has_block(uhd::rfnoc::block_id_t(std::to_string(qFloor(i/2)) + "/PSD_" + std::to_string(i % 2))))
    {
        qInfo() << endl << "PSD-block " + QString::number(i % 2) + " does not exist on device " + QString::number(qFloor(i/2)) + "!" << endl;
        throw "PSD-block " + QString::number(i % 2) + " does not exist on device " + QString::number(qFloor(i/2)) + "!";
    }

    uhd::rfnoc::source_block_ctrl_base::sptr psd_block_ctrl = usrpDevice->get_block_ctrl<uhd::rfnoc::source_block_ctrl_base>(std::to_string(qFloor(i/2)) + "/PSD_" + std::to_string(i % 2));
    rx_graph->connect(ddc_block_ctrl->get_block_id(), 0, psd_block_ctrl->get_block_id(), 0, 1);   //connect ddc with psd

    streamer_args["block_id" + std::to_string(i)] = psd_block_ctrl->get_block_id().to_string();    //connects psd to stream
    streamer_args["block_port" + std::to_string(i)] = "0";
    rx_channel_nums.push_back(i);
}

samps_per_buff = _ip->nfft * _ip->downsamplingFactor;
this->createNewBuffers();   //prepare buffer for streamed data

//Create a receive streamer:
uhd::stream_args_t stream_args("sc16", "sc16");
stream_args.channels = rx_channel_nums;
stream_args.args = streamer_args;
stream_args.args["spp"] = boost::lexical_cast<std::string>(256);
rx_stream = usrpDevice->get_rx_stream(stream_args);

qInfo() << endl << "Begin continuous streaming in " << QString::number(_ip->streamStartDelay * multiUSRP->get_num_mboards()) << " seconds...";

stream_cmd = QSharedPointer<uhd::stream_cmd_t>(new uhd::stream_cmd_t(uhd::stream_cmd_t::STREAM_MODE_START_CONTINUOUS));

stream_cmd->stream_now = false; //controls, when the stream begins. When false, the device will begin streaming at a time specified by time_spec
stream_cmd->time_spec = uhd::time_spec_t(_ip->streamStartDelay * multiUSRP->get_num_mboards());
rx_stream->issue_stream_cmd(*stream_cmd); //tells all channels to stream

Additional Information

I've wrote already in the mailing-list and it seems, I'm not the only person who has this problem!

[Kuurusch]

So someone helped me in the mailing-list and I could solve the problem for me with using the python-script under firmware/usrp3/x300/x300_debug.py --addr 192.168.10.1 --poke=0x00100058 --data=1

But I wouldn't say the issue is solved, its clearly a bug and during setup-process of USRP some buffers are not cleared!

It also didn't helped to wait for several minutes before restart. I think the problem is because after killing the app the streaming on the USRP continuous until I restart the app (led under RX2 port is still on).

[Kuurusch]

Would it be helpful to post our FPGA-Code concerning the PSD-block?

[Kuurusch]

I just forgot to mention, that we've tested our FPGA-Image firstly with GNU-Radio and there it works without a problem. But I think GNU-Radio does not use multi_usrp, it uses directly a device3, right?

[Kuurusch]

Just tested the final release of 3.14.0.0 and the problem is still there!

[natetemple]

@Kuurusch Can you please email support@ettus.com and we can help further debug the issue through that channel.