Stack stucks in LORAMAC_TX_RUNNING state if RX1 error occurs

iotfreaks commented 5 years ago

Hi! this issue was created upon @djaeckle request in #755 issue

If Rx error occurs during reception of packet in RX1 the stack stucks in LORAMAC_TX_RUNNING state.

Analysis: If RX1 error occurs, bit MacCtx.MacFlags.Bits.MacDone is never set to 1 (not sure why, but as is). The LORAMAC_TX_RUNNING is dropped inside StopRetransmission() function, that called inside LoRaMacHandleMcpsRequest(). But if MacDone is never set to 1, the LoRaMacProcess() function never calls LoRaMacHandleMcpsRequest().

void LoRaMacProcess( void )
{
    uint8_t noTx = false;

    LoRaMacHandleIrqEvents( );
    LoRaMacClassBProcess( );

    // MAC proceeded a state and is ready to check
    if( MacCtx.MacFlags.Bits.MacDone == 1 )
    {
        LoRaMacEnableRequests( LORAMAC_REQUEST_HANDLING_OFF );
        LoRaMacCheckForRxAbort( );

        // An error occurs during transmitting
        if( IsRequestPending( ) > 0 )
        {
            noTx |= LoRaMacCheckForBeaconAcquisition( );
        }

        if( noTx == 0x00 )
        {
            LoRaMacHandleMlmeRequest( );
            LoRaMacHandleMcpsRequest( );
        }
        LoRaMacHandleRequestEvents( );
        LoRaMacHandleScheduleUplinkEvent( );
        LoRaMacEnableRequests( LORAMAC_REQUEST_HANDLING_ON );
    }
    LoRaMacHandleIndicationEvents( );
    if( MacCtx.RxSlot == RX_SLOT_WIN_CLASS_C )
    {
        OpenContinuousRxCWindow( );
    }
}

To simulate this problem, you need to place device in harsh radio environement with RSSI and SNR close to demodulation floor.

We make a night-test and problem is gone. This patch solves the problem for us:

static void HandleRadioRxErrorTimeout( LoRaMacEventInfoStatus_t rx1EventInfoStatus, LoRaMacEventInfoStatus_t rx2EventInfoStatus )
{
    bool classBRx = false;

    if( MacCtx.NvmCtx->DeviceClass != CLASS_C )
    {
        Radio.Sleep( );
    }

    if( LoRaMacClassBIsBeaconExpected( ) == true )
    {
        LoRaMacClassBSetBeaconState( BEACON_STATE_TIMEOUT );
        LoRaMacClassBBeaconTimerEvent( NULL );
        classBRx = true;
    }
    if( MacCtx.NvmCtx->DeviceClass == CLASS_B )
    {
        if( LoRaMacClassBIsPingExpected( ) == true )
        {
            LoRaMacClassBSetPingSlotState( PINGSLOT_STATE_CALC_PING_OFFSET );
            LoRaMacClassBPingSlotTimerEvent( NULL );
            classBRx = true;
        }
        if( LoRaMacClassBIsMulticastExpected( ) == true )
        {
            LoRaMacClassBSetMulticastSlotState( PINGSLOT_STATE_CALC_PING_OFFSET );
            LoRaMacClassBMulticastSlotTimerEvent( NULL );
            classBRx = true;
        }
    }
    /** ---------------------------------------Added-------------------------------*/
    if(rx1EventInfoStatus == LORAMAC_EVENT_INFO_STATUS_RX1_ERROR)  
    {                                                                                                                
      MacCtx.MacFlags.Bits.MacDone = 1;                                                       
    }    
    /**-------------------------------------------------------------------------------*/                                                                                                              
    if( classBRx == false )
    {
        if( MacCtx.RxSlot == RX_SLOT_WIN_1 )
        {
            if( MacCtx.NodeAckRequested == true )
            {
                MacCtx.McpsConfirm.Status = rx1EventInfoStatus;
            }
            LoRaMacConfirmQueueSetStatusCmn( rx1EventInfoStatus );
        }
        else
        {
            if( MacCtx.NodeAckRequested == true )
            {
                MacCtx.McpsConfirm.Status = rx2EventInfoStatus;
            }
            LoRaMacConfirmQueueSetStatusCmn( rx2EventInfoStatus );

            if( MacCtx.NvmCtx->DeviceClass != CLASS_C )
            {
                MacCtx.MacFlags.Bits.MacDone = 1;
            }
        }
    }

    UpdateRxSlotIdleState( );
}

ghost commented 5 years ago

unsubscribe me, please

iotfreaks notifications@github.com 于2019年7月4日周四下午4:20写道：

Hi! this issue was created upon @djaeckle https://github.com/djaeckle request in #755 https://github.com/Lora-net/LoRaMac-node/issues/755 issue

If Rx error occurs during reception of packet in RX1 the stack stucks in LORAMAC_TX_RUNNING state.

Analysis: If RX1 error occurs, bit MacCtx.MacFlags.Bits.MacDone is never set to 1 (not sure why, but as is). The LORAMAC_TX_RUNNING is dropped inside StopRetransmission() function, that called inside LoRaMacHandleMcpsRequest(). But if MacDone is never set to 1, the LoRaMacProcess() function never calls LoRaMacHandleMcpsRequest().

void LoRaMacProcess( void ) { uint8_t noTx = false;
LoRaMacHandleIrqEvents( );
LoRaMacClassBProcess( );

// MAC proceeded a state and is ready to check
if( MacCtx.MacFlags.Bits.MacDone == 1 )
{
    LoRaMacEnableRequests( LORAMAC_REQUEST_HANDLING_OFF );
    LoRaMacCheckForRxAbort( );

    // An error occurs during transmitting
    if( IsRequestPending( ) > 0 )
    {
        noTx |= LoRaMacCheckForBeaconAcquisition( );
    }

    if( noTx == 0x00 )
    {
        LoRaMacHandleMlmeRequest( );
        LoRaMacHandleMcpsRequest( );
    }
    LoRaMacHandleRequestEvents( );
    LoRaMacHandleScheduleUplinkEvent( );
    LoRaMacEnableRequests( LORAMAC_REQUEST_HANDLING_ON );
}
LoRaMacHandleIndicationEvents( );
if( MacCtx.RxSlot == RX_SLOT_WIN_CLASS_C )
{
    OpenContinuousRxCWindow( );
}
}

To simulate this problem, you need to place device in harsh radio environement with RSSI and SNR close to demodulation floor.

We make a night-test and problem is gone. This patch solves the problem for us:

static void HandleRadioRxErrorTimeout( LoRaMacEventInfoStatus_t rx1EventInfoStatus, LoRaMacEventInfoStatus_t rx2EventInfoStatus ) { bool classBRx = false;
if( MacCtx.NvmCtx->DeviceClass != CLASS_C )
{
    Radio.Sleep( );
}

if( LoRaMacClassBIsBeaconExpected( ) == true )
{
    LoRaMacClassBSetBeaconState( BEACON_STATE_TIMEOUT );
    LoRaMacClassBBeaconTimerEvent( NULL );
    classBRx = true;
}
if( MacCtx.NvmCtx->DeviceClass == CLASS_B )
{
    if( LoRaMacClassBIsPingExpected( ) == true )
    {
        LoRaMacClassBSetPingSlotState( PINGSLOT_STATE_CALC_PING_OFFSET );
        LoRaMacClassBPingSlotTimerEvent( NULL );
        classBRx = true;
    }
    if( LoRaMacClassBIsMulticastExpected( ) == true )
    {
        LoRaMacClassBSetMulticastSlotState( PINGSLOT_STATE_CALC_PING_OFFSET );
        LoRaMacClassBMulticastSlotTimerEvent( NULL );
        classBRx = true;
    }
}
/** ---------------------------------------Added-------------------------------*/
if(rx1EventInfoStatus == LORAMAC_EVENT_INFO_STATUS_RX1_ERROR)
{
  MacCtx.MacFlags.Bits.MacDone = 1;
}
/**-------------------------------------------------------------------------------*/
if( classBRx == false )
{
    if( MacCtx.RxSlot == RX_SLOT_WIN_1 )
    {
        if( MacCtx.NodeAckRequested == true )
        {
            MacCtx.McpsConfirm.Status = rx1EventInfoStatus;
        }
        LoRaMacConfirmQueueSetStatusCmn( rx1EventInfoStatus );
    }
    else
    {
        if( MacCtx.NodeAckRequested == true )
        {
            MacCtx.McpsConfirm.Status = rx2EventInfoStatus;
        }
        LoRaMacConfirmQueueSetStatusCmn( rx2EventInfoStatus );

        if( MacCtx.NvmCtx->DeviceClass != CLASS_C )
        {
            MacCtx.MacFlags.Bits.MacDone = 1;
        }
    }
}

UpdateRxSlotIdleState( );
}

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Lora-net/LoRaMac-node/issues/757?email_source=notifications&email_token=AC65R4ZI5YCP5WNFUU6ZNITP5XIY5A5CNFSM4H5YAAMKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G5LEKSA, or mute the thread https://github.com/notifications/unsubscribe-auth/AC65R47VXX7KBVE3ZVDEATDP5XIY5ANCNFSM4H5YAAMA .

djaeckle commented 5 years ago

Hi @iotfreaks,

thanks for creating the issue. Which version and region are you using?

iotfreaks commented 5 years ago

Hi @djaeckle . Develop branch with latest commits. Regions EU-868, RU-864, it not depends from region for us.

branek commented 5 years ago

@iotfreaks thank you.

We are able to confirm same bug (any region, feature/5.0.0 branch).

We tried your proposed solution (it works), but maybe it's better to put code under if statement which handles RX_SLOT_WIN_1 reception:

static void HandleRadioRxErrorTimeout( LoRaMacEventInfoStatus_t rx1EventInfoStatus, LoRaMacEventInfoStatus_t rx2EventInfoStatus )
{
    bool classBRx = false;

    if( MacCtx.NvmCtx->DeviceClass != CLASS_C )
    {
        Radio.Sleep( );
    }

    if( LoRaMacClassBIsBeaconExpected( ) == true )
    {
        LoRaMacClassBSetBeaconState( BEACON_STATE_TIMEOUT );
        LoRaMacClassBBeaconTimerEvent( NULL );
        classBRx = true;
    }
    if( MacCtx.NvmCtx->DeviceClass == CLASS_B )
    {
        if( LoRaMacClassBIsPingExpected( ) == true )
        {
            LoRaMacClassBSetPingSlotState( PINGSLOT_STATE_CALC_PING_OFFSET );
            LoRaMacClassBPingSlotTimerEvent( NULL );
            classBRx = true;
        }
        if( LoRaMacClassBIsMulticastExpected( ) == true )
        {
            LoRaMacClassBSetMulticastSlotState( PINGSLOT_STATE_CALC_PING_OFFSET );
            LoRaMacClassBMulticastSlotTimerEvent( NULL );
            classBRx = true;
        }
    }

    if( classBRx == false )
    {
        if( MacCtx.RxSlot == RX_SLOT_WIN_1 )
        {
            if( MacCtx.NodeAckRequested == true )
            {
                MacCtx.McpsConfirm.Status = rx1EventInfoStatus;
            }
            LoRaMacConfirmQueueSetStatusCmn( rx1EventInfoStatus );
/** ---------- is this right place for fix ----------*/
            if(rx1EventInfoStatus == LORAMAC_EVENT_INFO_STATUS_RX1_ERROR)
            {
              MacCtx.MacFlags.Bits.MacDone = 1;
            }
/** -------------------- end of fix -------------------*/
        }
        else
        {
            if( MacCtx.NodeAckRequested == true )
            {
                MacCtx.McpsConfirm.Status = rx2EventInfoStatus;
            }
            LoRaMacConfirmQueueSetStatusCmn( rx2EventInfoStatus );

            if( MacCtx.NvmCtx->DeviceClass != CLASS_C )
            {
                MacCtx.MacFlags.Bits.MacDone = 1;
            }
        }
    }

    UpdateRxSlotIdleState( );
}

iotfreaks commented 5 years ago

Hi @branek ! Currently, RX errors not handled inside this function. Not RX1 not RX2 for any class. So we assume, that bugfix should be little bit more complex than we or you offer. We hope @djaeckle can help understand what is happening under the hood).

djaeckle commented 5 years ago

Hi all,

thanks for your reports and the participation. In previous versions we had a verification regarding this issue. Here is the proposal:

static void HandleRadioRxErrorTimeout( LoRaMacEventInfoStatus_t rx1EventInfoStatus, LoRaMacEventInfoStatus_t rx2EventInfoStatus )
{
    bool classBRx = false;

    if( MacCtx.NvmCtx->DeviceClass != CLASS_C )
    {
        Radio.Sleep( );
    }

    if( LoRaMacClassBIsBeaconExpected( ) == true )
    {
        LoRaMacClassBSetBeaconState( BEACON_STATE_TIMEOUT );
        LoRaMacClassBBeaconTimerEvent( NULL );
        classBRx = true;
    }
    if( MacCtx.NvmCtx->DeviceClass == CLASS_B )
    {
        if( LoRaMacClassBIsPingExpected( ) == true )
        {
            LoRaMacClassBSetPingSlotState( PINGSLOT_STATE_CALC_PING_OFFSET );
            LoRaMacClassBPingSlotTimerEvent( NULL );
            classBRx = true;
        }
        if( LoRaMacClassBIsMulticastExpected( ) == true )
        {
            LoRaMacClassBSetMulticastSlotState( PINGSLOT_STATE_CALC_PING_OFFSET );
            LoRaMacClassBMulticastSlotTimerEvent( NULL );
            classBRx = true;
        }
    }

    if( classBRx == false )
    {
        if( MacCtx.RxSlot == RX_SLOT_WIN_1 )
        {
            if( MacCtx.NodeAckRequested == true )
            {
                MacCtx.McpsConfirm.Status = rx1EventInfoStatus;
            }
            LoRaMacConfirmQueueSetStatusCmn( rx1EventInfoStatus );
// START --------------------------------------------------
            if( TimerGetElapsedTime( MacCtx.NvmCtx->LastTxDoneTime ) >= MacCtx.RxWindow2Delay )
            {
                TimerStop( &MacCtx.RxWindowTimer2 );
                MacCtx.MacFlags.Bits.MacDone = 1;
            }
// END --------------------------------------------------
        }
        else
        {
            if( MacCtx.NodeAckRequested == true )
            {
                MacCtx.McpsConfirm.Status = rx2EventInfoStatus;
            }
            LoRaMacConfirmQueueSetStatusCmn( rx2EventInfoStatus );

            if( MacCtx.NvmCtx->DeviceClass != CLASS_C )
            {
                MacCtx.MacFlags.Bits.MacDone = 1;
            }
        }
    }

    UpdateRxSlotIdleState( );
}

However, please note that i was not able to test it yet. Also, we need to verify it on the radio level.

branek commented 4 years ago

We are able to confirm this fix is working as expected when device tested for several days.

AliceWribbit commented 4 years ago

Hi @djaeckle ! I actually faced the same problem that Stack stucks in LORAMAC_TX_RUNNING. I fix the problem with your code in USB charging mode. If i switch to battery mode, the problem happen again.It seemed that the problem haven't solved in low power mode. My hardware playform is NucleoL073 with SX1278. In usb charging mode (bool UsbIsConnected = true;),i test it for 7days, no problem. In battery mode(bool UsbIsConnected = false;),the problem appear easily especially when cutting off internet access of gateway after node OTAA joined.

djaeckle commented 4 years ago

Hi @AliceWribbit,

thanks for your report. I would be nice if you could open up a new issue with a reference to this one and a description of your error case. Thanks in advance!

EDIT: Just saw that you have already done that.

AliceWribbit commented 4 years ago

Hi @djaeckle , i opened up a new issue in #832

Lora-net / LoRaMac-node

Stack stucks in LORAMAC_TX_RUNNING state if RX1 error occurs #757