search

nthallen / monarch

Monarch Data Acquisition System
0 stars 1 forks source link

TMC: int32 overflow in generated calibration #204

Closed nthallen closed 9 months ago

nthallen commented 10 months ago 
TM typedef int16_t B3MB_Volts_t { text "%7.3lf"; Convert B3MB_VOLTS; }
TM typedef B3MB_Volts_t B3MB_100V3_B1V_t;
TM 1 Hz B3MB_100V3_B1V_t  B3MB_100V3_Batt1_V;
Calibration (B3MB_100V3_B1V_t, B3MB_VOLTS) {
      0,    0.118,
  32768,  206.551
}

Generates:

/* int icvt for B3MB_100V3_B1V_t -> B3MB_VOLTS */
static int32_t _CVT_45( B3MB_100V3_B1V_t x) {
  int32_t ov;

  if (x < -15777) {
    ov = (x*(3824L)+125305135)/(607)-206315;
  } else {
    if (x < 16991) {
      ov = (x*(3824L)+60331854)/(607)-99275;
    } else {
      ov = (x*(133777L)+2021983515)/(21235)+107158;
    }
  }
  return ov;
}

and the quantity (x*133777L)+2021983515, when evaluated for x = 32768 yields something around 6.4e9.

nthallen commented 10 months ago

This is mentioned in #136