HTM Scale-up Not Having The Effect Expected

[tbhallett]

Whereas, the test script and the initial scenario run showed that the HTM scale-up was working as expected, it appears that the in practice they do not (see here).

The test script and the scenario both examine scale-up occurring instantly in year 2012, whereas the scale-up scenarios we want for the final analysis is for scale-up instantly in 2019.

Looking at this, I can see a few reasons why this might be occurring:

• We write to a parameter that does not exist: https://github.com/UCL/TLOmodel/blob/e9263803787825168955b0e24ed15c383a2ba0dd/src/tlo/methods/hiv.py#L1131-L1132

• Updating of the rates of testing does not work as expected, as the values for years after 2020 are set to nan, and note that future years are never read due to this logic. https://github.com/UCL/TLOmodel/blob/e9263803787825168955b0e24ed15c383a2ba0dd/src/tlo/methods/hiv.py#L1124-L1125

(I don't think this section suffers from the same issue - but it looks a bit fragile, for similar reasons).

• We write new values for parameter values, but we do not rebuild the LinearModels that use them (the linear model stores the value of parameter when it is built, and does not update when those parameters are updated later) https://github.com/UCL/TLOmodel/blob/e9263803787825168955b0e24ed15c383a2ba0dd/src/tlo/methods/hiv.py#L1113C13-L1113C52 https://github.com/UCL/TLOmodel/blob/e9263803787825168955b0e24ed15c383a2ba0dd/src/tlo/methods/hiv.py#L1121

There may be other issues in the Tb and malaria codes of a similar nature.

I would propose doing the following:

• [ ] Update the scenario file so that it reflects a scale-up in 2019, running at scale, and confirming that this also shows there being no epidemiological impact. (If this doesn't confirm it, then I have had some mistakes in my Scenario file and in these observations)
• [ ] Searching for possible issues in hiv.py, tb.py and malaria.py and fixing
• [ ] Re-running the scenario
• [ ] Raising PR with fixes

[tdm32]

Whereas, the test script and the initial scenario run showed that the HTM scale-up was working as expected, it appears that the in practice they do not (see here).

The test script and the scenario both examine scale-up occurring instantly in year 2012, whereas the scale-up scenarios we want for the final analysis is for scale-up instantly in 2019.

Looking at this, I can see a few reasons why this might be occurring:

• We write to a parameter that does not exist: https://github.com/UCL/TLOmodel/blob/e9263803787825168955b0e24ed15c383a2ba0dd/src/tlo/methods/hiv.py#L1131-L1132
• Updating of the rates of testing does not work as expected, as the values for years after 2020 are set to nan, and note that future years are never read due to this logic. https://github.com/UCL/TLOmodel/blob/e9263803787825168955b0e24ed15c383a2ba0dd/src/tlo/methods/hiv.py#L1124-L1125

(I don't think this section suffers from the same issue - but it looks a bit fragile, for similar reasons).

• We write new values for parameter values, but we do not rebuild the LinearModels that use them (the linear model stores the value of parameter when it is built, and does not update when those parameters are updated later) https://github.com/UCL/TLOmodel/blob/e9263803787825168955b0e24ed15c383a2ba0dd/src/tlo/methods/hiv.py#L1113C13-L1113C52 https://github.com/UCL/TLOmodel/blob/e9263803787825168955b0e24ed15c383a2ba0dd/src/tlo/methods/hiv.py#L1121

There may be other issues in the Tb and malaria codes of a similar nature.

I would propose doing the following:

• [ ] Update the scenario file so that it reflects a scale-up in 2019, running at scale, and confirming that this also shows there being no epidemiological impact. (If this doesn't confirm it, then I have had some mistakes in my Scenario file and in these observations)
• [ ] Searching for possible issues in hiv.py, tb.py and malaria.py and fixing
• [ ] Re-running the scenario
• [ ] Raising PR with fixes

Thanks for digging into this @tbhallett, I'll work on it this afternoon. Just confirming branch [hallett/analysis_for_vertical_and_horizontal_progs] is the most up-to-date one?

[tdm32]

@tbhallett I'm wondering if there is an issue here with constraints on consumables and personnel. From my scale-up paper, there was only a small impact of HIV program scale-up for 2 reasons: 1. the program is already very close to hitting targets and the proposed scale-up is small, 2. the consumables availability limits the amount of scale-up that can occur.

I've tested this quickly on a small population and found that running the scenarios with consumables availability set to 'all' did lead to a reduction in AIDS deaths with HIV program scale-up (around 30% fall in a pop of 5,000).

So when we scale-up programs only (no HSS), the effects will be limited by cons availability. Both the baseline and the scale-up scenario will likely be affected by mode 2, I think in mode 2 even in the baseline scenarios many appts will not be able to run due to personnel shortages.

Could this be explaining why we see no difference between the baseline and scale-up scenarios? I'll set off a larger run with cons availability to 'all' and keep mode 1 throughout just to see. Hopefully will update you Thurs am.

[tbhallett]

@tbhallett I'm wondering if there is an issue here with constraints on consumables and personnel. From my scale-up paper, there was only a small impact of HIV program scale-up for 2 reasons: 1. the program is already very close to hitting targets and the proposed scale-up is small, 2. the consumables availability limits the amount of scale-up that can occur.

I've tested this quickly on a small population and found that running the scenarios with consumables availability set to 'all' did lead to a reduction in AIDS deaths with HIV program scale-up (around 30% fall in a pop of 5,000).

So when we scale-up programs only (no HSS), the effects will be limited by cons availability. Both the baseline and the scale-up scenario will likely be affected by mode 2, I think in mode 2 even in the baseline scenarios many appts will not be able to run due to personnel shortages.

Could this be explaining why we see no difference between the baseline and scale-up scenarios? I'll set off a larger run with cons availability to 'all' and keep mode 1 throughout just to see. Hopefully will update you Thurs am.

That might be a part of it, but not all (I think), because;

• There is no difference between scenario B (massive healthsystem strengthening) and J (massive health system stqrenghenin plus HTM scale-up)
• Some of the changes in the scale-up should be "free" of cost for the healthsystem, e.g. changes in beta.

[tbhallett]

We made the change whereby the linear models are rebuild following updating the parameters in #1413

The graphs produced were almost the same as the originals above.

[tbhallett]

Next job is to run a simplified version of the analysis, but...

• In Mode 1 (so we can see if it is the "mode 2 constraints" that is hampering the scale-up
• With the logs from HIV, TB and Malaria modules turned on, so that we can see if the service coverages are increasing as expected.

This is being done on within the branch PR'd at https://github.com/UCL/TLOmodel/pull/1288 in file src/scripts/comparison_of_horizontal_and_vertical_programs/mini_analysis_for_testing/mini_version_scenario.py

~job-id = htm_with_and_without_hss-2024-07-25T193651Z~

UPDATE:

That job failed for mysterious reason. Second attempt is (from current tip of master c147e2bfbf715e262f38411269776582d14273a4)

mini_htm_with_and_without_hss-2024-07-29T102041Z

[tbhallett]

@tdm has done experiments that confirm ---for a scale-up in 2015, to "maximal possible levels", in either Mode 1 or Mode 2, an effect on deaths is had for H, T, M scale-ups individually and collectively.

Therefore, the discrepancy between this success and the lack of an effect not the runs done so far using the Scenario files defined above could be as follows:

• The scale-up being in 2015 as opposed to 2019
• The scale-up being "max" in Tara's runs (vs. something slightly short of max) in the Scenario runs.
• Something to do with the differences in coding between the two runs (The Scenario file including *rescaling to Mode 2, scaling due to absences etc etc.)

Therefore, our action plan is as follows (in some new PRs)

• Tara to bring in the "max" variant of the scenarios so that we can switch to using these. [#1441]
• Re-run using max scenarios AND 2015 as the start date of the switching
• Remove rescaling due to staff absences (now that we've moved away from doing that). [#1439]

And also,

• Re-run the "mini" version of the analysis (https://github.com/UCL/TLOmodel/issues/1411#issuecomment-2250851331) to see if running just the HTM scale-up in Mode 1 has an effect.

[tdm32]

Below are test runs which use default settings for all modules and switch the HIV, TB and malaria programmes to minimum or maximum values. All runs are done in both mode 1 and mode 2.

TO DO:

• create new switch so programmes can be scaled to either programme targets or max values
• merge new switch and perform test runs with different HS settings to identify any issues potentially mitigating HTM programme impact

with services set to MIN:

with services set to MAX:

[tbhallett]

Following merge of #1443 and #1441,

we're now submitting a run using the MAX scenario and with changes occurring in 2015.

This is from branch hallett/test_run_htm_scale_up and the job_id is: htm_with_and_without_hss-2024-07-29T143852Z

[tbhallett]

Next steps

• ~[x] examine the mini-run that used Mode 1 (but before all the fixes came in)~ IGNORE: I had set this up wrongly. Trying again!
• [x] examine the 'full' run (at small scale) (see below)

[tbhallett]

Results from the full_run look more promising....

Need to work out is 2019 scale-up can work too and if it's reasonable to use the max scale-ups.

Check that the HTM input is of the same size as Tara expects (from her earlier paper).

Update graphs. (It could be that the effects look small because the HSS package is so strong.)

Also - effect on TB very small. Maybe best to combine HIV and TB for presentation purposes?

[tbhallett]

final attempt at running the whole thing in Mode1 : htm_with_and_without_hss_mode1-2024-08-02T160157Z (which is a run of src/scripts/comparison_of_horizontal_and_vertical_programs/scenario_vertical_programs_with_and_without_hss_mode1.py

[tbhallett]

next steps:

• [ ] look at the mode 1 results
• [ ] examine the mode2 and mode1 results to see what's happening to the programs, and the HSIs
• [ ] is the muted impact of scale-up (especially TB and malaria) in mode 2 to do with crowding-out of HSI (lots of referrals and false positives, clogging-up everything?)
• [ ] should we say that there is an efficiency boost in HTM appointments, to reduce their footprints?

[tbhallett]

Results from mode1 run (htm_with_and_without_hss_mode1-2024-08-02T160157Z) seem very similar to that for Mode2. This suggests that the 'crowding-out' issues are not behind the weak impact of the HTM scale-up....?

[tbhallett]

@tdm's further investigation finds 1) Diagnosis rate for TB remains low in scale-up scenario (due to low sensivity of Gene Xpert on sputum samples) 2) That scale-up is frustrated by consumables non-availability to extreme degree 3) That in TB scale-up scenario, TB treatment rates goes down (not up)

• She has made a fix for (1) (https://github.com/UCL/TLOmodel/pull/1453) that means that persons that are suspected to have TB, but have a negative test, are referred for a culture test (with very high sensitivity). We're running with that from that branch (https://github.com/UCL/TLOmodel/pull/1453): job_id is htm_with_and_without_hss-2024-08-06T094337Z

• To determine if indeed, the limitations largely come from consumables (and that, conversely, most of the HSS package benefits come from removing that constraint), we'll run the whole analysis, forcing consumables to be always available: job_id is htm_with_and_without_hss-2024-08-06T094422Z

• On (3), @tdm32 is investigating....

[tdm32]

@tbhallett regarding issue 3 above, from plots of run htm_with_and_without_hss_mode1-2024-08-02T160157Z we saw unusual trends in tb proportion diagnosed and proportion treated.

Turns out this was just an anomaly in the final year of the simulation, whereby the proportion dx (and therefore treated) drops. This may just be stochastic noise, or a low proportion of tb cases occurring in PLHIV (who get additional screening compared with general population) 12% in the final year.

To correct this, I have re-plotted the figures using an average over the period 2016-2030. This shows almost identical treatment coverage rates between baseline and TB_no_HSS (both 38%), with increase to 43% with HSS. For diagnosis, TB_no_HSS shows small increase from 38% to 48%, 45% with HSS. No obvious reason why dx rates are slightly lower with HSS, but we do have small tb case numbers for this (80-100 cases per year).

Looks like almost perfect referral between dx and tx for TB+HSS, but some drop-off in TB_no_HSS which is expected due to consumables stockouts.

I'll repeat this with the new runs so we can see the impact of TB Culture HSI and supply chain strengthening alone vs full HSS.

TB

HIV

MALARIA

[tbhallett]

Results from htm_with_and_without_hss-2024-08-06T094337Z (using the TB fix for culture)

[tbhallett]

... and from htm_with_and_without_hss-2024-08-06T094422Z (where we also have the consumables always being available):

[tdm32]

From run htm_with_and_without_hss-2024-08-06T094422Z where consumables are always available and TB culture used as a backup dx test we do see an improvement in tb diagnosis and treatment rates: