Severson-Group / AMDC-Firmware

Embedded system code (C and Verilog) which runs the AMDC Hardware
http://docs.amdc.dev/firmware
BSD 3-Clause "New" or "Revised" License
31 stars 5 forks source link

Enabling/disabling sensors during operation leads to inconsistent scheduler tick loop time #402

Closed codecubepi closed 1 month ago

codecubepi commented 3 months ago

Plots of issue: disabling and re-enabling sensors during operation leads to inconsistent scheduler_tick() loop time:

image

image

See the conversation on #399 for full background.

codecubepi commented 2 months ago

Mode 0: Scheduler ISR is generated based on just the PWM carrier and User Ratio

Plots are shown for PWM 100 kHz and ratio of 10, for a control frequency of 10 kHz (expected time between Scheduler ISR calls is 100us)

Mode 1: With no sensors enabled, should function just like Mode 0 above. With sensors enabled, Scheduler ISR is generated after all sensors are done

These plots are shown for again, PWM 100 kHz and ratio of 10, for a control frequency of 10 kHz (expected time between Scheduler ISR calls is 100us). These are enabling and disabling the ADC. The previous comment in the conversation shows enabling and disabling the AMDS.

npetersen2 commented 2 months ago

@codecubepi huh, interesting.

I am wondering if the 10 usec interval when the sensors are disabled is actually just the PWM carrier period?

Can you run another test where you set the PWM carrier to 200 kHz and the ratio such that the control rate is still 10 kHz? Then, show the same plots and see if the low part of the signal goes down to 5 usec.

codecubepi commented 2 months ago

Enabling/disabling AMDS in timing manager in all cases:

PWM 100 kHz, Ratio 10

image

PWM 200 kHz, Ratio 20

image

PWM 150 kHz, Ratio 15

image

Wacky... so it seems that the bad scheduler frequency is not directly correlated with the PWM/ratio settings. But there is definitely something wrong in all cases only when all sensors are disabled.

Again, I wasn't able to observe this behaviour in my simulation.

codecubepi commented 1 month ago

Progress!

I made some (what should be) meaningless FPGA changes that may have fixed the problem... apparently Vivado's elaboration and synthesis of combination logic controlling toggling a flop is odd.

Changes made (7e88b4e)

image

Screenshots of Results

PWM 100 kHz, Ratio 10

PWM 200 kHz, Ratio 20

PWM 150 kHz, Ratio 15

Unfortunately: the loop time for the control task is double what it should be, which results in the output load frequency being halved (I am seeing 300 Hz instead of 600 Hz on the uInverter/scope bench).

codecubepi commented 1 month ago

I was able to band-aid fix the problem with the doubled loop time and halved output frequency by adding additional tolerance when deciding to schedule the task in scheduler_run(). This seems to be due to the floating point inaccuracies seen before. For this particular frequency set, I had to increase the SCHEDULER_INTERVAL_TOLERANCE_USEC from -0.06 (60 ns) to -0.15 (150 ns)... which is quite a lot.

codecubepi commented 1 month ago

Results for the 150 kHz PWM, Ratio 15 case, with 150ns SCHEDULER_INTERVAL_TOLERANCE_USEC here:

Note that this is still with no event-synchronized latching of the sensor enable signals. Also... this whole idea of enabling and disabling sensors while the control task is running is pretty unlikely to come up anyway. So, good to go?

codecubepi commented 1 month ago

See commit 8db3f4d, which adds a convenient way for the user to override the scheduler tolerance value if abnormal timing settings necessitate doing so.