Debugging errors in dismantlement

The problem: In dismantlement of Portugal, non-organized territories do not seem to be handled properly.

Evidence: The event happens repeatedly as shown in this screenshot, although colonies such as Guinea-Bissau and Mozambique are not released and are not further dealt with:

Debugging process: Since I am mostly unfamiliar with exactly what will happen in this particular dismantlement, I want to be very cautious before making assumptions. For this reason, I wanted to track down the exact dismantlement event that will be triggered. In this case, I guessed that it would be 96010, since that seems to be potentially generic dismantlement for non-great powers as well, and this seems to be correct:

Looking through that event, I think this section should have released the Portuguese colonies that were not released:

https://github.com/moretrim/ccHFM/blob/47becfb5ccb5c1e4786bb500f718b524df991b20/ccHFM/events/GreatWar_Events.txt#L978-L990

For some reason, the following does not seem to be working as expected for Guinea-Bissau and Mozambique, but it works for Angola. This suggests that either the code cannot release these tags, or the above trigger does not work for Guinea-Bissau and Mozambique (more likely in my estimation).

https://github.com/moretrim/ccHFM/blob/47becfb5ccb5c1e4786bb500f718b524df991b20/ccHFM/events/GreatWar_Events.txt#L11349-L11356

I think any_core = {owned_by = THIS } and exists = no are satisfied. Looking at cultures.txt, I see no evidence of a cultural union involving either. Therefore, I am left wondering if capital_scope is not working.

The involved tags and capitals: 2061 is the capital of AGC, Angoche, owned by POR. 1878 is the capital of GBU, Gabu/Guinea-Bissau, owned by POR. 2058 is the capital of GAZ, Gaza, owned by POR. 2070 is the capital of SHO Shona, is not owned by POR, and therefore is potentially not involved. It is not clear what would happen with provinces that only have SHO cores. It is also possible that I misunderstand the role of this.

The next question is this: do AGC, GAZ, and GBU have unciv_tech_school?

After a while of testing, I am not so sure how capital_scope is working here. For instance, an event with trigger = { capital_scope = { owner = { tech_school = unciv_tech_school } } } does not work. Likewise, a decision with similar wording does not work.

I will keep testing to see if I can get it to work the way that it looks like it should work.

Further tests:

Angoche, Gaza, Guinea-Bissau, and Shona are all releasable tags in the save that experiences the problem.
If I remove the capital_scope that is indicated as a potential issue above, does dismantlement work? Yes.
Why did Angola work? It is scoped differently, since it has the flag post_colonial_country. It need only have capital_scope = { owned_by = THIS } function, without the added requirement of checking the tech school.
Does it work if I substitute civilized = no for the potentially broken scope? Yes.
Does it work if I add capital_scope = { owned_by = THIS } to it again after the above test? Partially. Angoche and Gaza work. Shona and Guinea-Bissau do not work, presumably because their capitals are not owned by Portugal (despite what I posted about GBU above).
Does this fix cause problems with other generic dismantlements? That will require some testing.

I intend to expand this to examination of dismantlement in other forms. it looks like the above issue also takes place when dismantling the UK, which is a separate process. I plan to make the above change to all the separate dismantlements. In addition, the following issues occur with UK dismantlement:

Colonies without cores do not dismantle. This can include Cape Verde from Portugal, for instance.
The Suez Canal will not dismantle.

It seems that some or all of these bugs could be due to syntax errors such as that described for Portugal above. Correcting this error for UK dismantlement allowed Burundi, Ceylon, Malaysia, and Pakistan to dismantle correctly. Further fixes may allow the others to dismantle correctly.

The proposed mechanism for these bugs is that when a scope is used in a way that the engine cannot parse, it may jump ahead until it starts parsing code correctly again. This may cause the event to not function as intended.

Additionally, the dismantlement process cannot award a landlocked satellite. This is why Nyasaland cannot be awarded in the UK test save that I have. If I am awaiting dismantlement of Kenya, Uganda, Burundi, and Rwanda, and I release Kenya before the others are awarded, the others will never be awarded to anyone who does not border the satellite.

Since this is complex, it seems like a great idea to make a map, so that I don't keep checking the same things repeatedly:

being_dismantled: This is a country modifier whose purpose is to be a gatekeeper for dismantlement events. If this modifier is active while a country has at least one satellite, dismantlement continues.

colony_to_be_annexed: This is a country modifier whose purpose is to govern dismantlement rounds, functioning as a timer. If a country has this modifier, is a vassal, is ai, and does not have the modifier considering_colonial_offer while a dismantlement is occurring, it causes the country's overlord to experience event 70005 to start a new round of dismantlement through the event chain that continues with 96095. The localisation for this modifier is Colonial Negotiations.

colony_to_be_annexed_flag: This is a country flag, whose purpose is to release colonies that were not annexed during dismantlement. If a satellite has this flag and does not have colony_to_be_annexed, it is released.

considering_colonial_offer: This is a country modifier that is added with a short duration (2 days). It is added only in event 96095. If it is absent, but colony_to_be_annexed is present and the country has a port, it is used to call the aforementioned event to have it added. It seems to be a gatekeeper for a country that is dismantling another country. Once this modifier is present, 96096 is called to ask whether the dismantling country wants territory from the dismantlement or not. Then, 96097 is called to ask a (former) satellite if it will submit to annexation or not. In other words, a country cannot take another colony while it is considering one already.

Trying to explain it in a more lucid way here, using the definitions above:

Decision new_round_of_dismantlement requires colony_to_be_annexed, is_vassal and a dismantlement. The vassal cannot have considering_colonial_offer for this, because it would not make sense to petition the overlord to distribute it more than one war victor. Therefore, satellites (multiple ones at the same time) ask their overlord to give them away during dismantlement.
Event 70005 (" $COUNTRY_ADJ$ Dismantlement") is next, on the overlord of the above vassal. This event picks a vassal that has the modifier colony_to_be_annexed, which does not have considering_colonial_offer, and which has at least one port.
Event 96095 ("The Fate of...") is called for the picked vassal from 70005. It sets the colony_to_be_annexed modifier and colony_to_be_annexed_flag for this vassal. It adds considering_colonial_offer for a random amount of time, preventing a repeat of the above steps for that vassal over that time. It then leads to the next step for a random greater or secondary power war victor with a truce. Therefore, if a vassal has the markers from above, and war victors never get the next step for it, then the problem is likely to be here.
Event 96096 ("The $FROMCOUNTRY_ADJ$ Colony") assigns infamy for the distributed vassal and asks the war victor whether it wants the colony or not.
Event 96097 ("The $FROMCOUNTRY_ADJ$ Ultimatum") is called for the former vassal from previous steps, asking whether it will submit or fight.

So what is happening with Nyasaland in this? It turns out to be multiple things:

It cannot function in my copy of 70005, because it does not have any ports. Note that master does not have that:

https://github.com/moretrim/ccHFM/blob/47becfb5ccb5c1e4786bb500f718b524df991b20/ccHFM/events/OnAction.txt#L373-L383

I don't know why I had total_num_of_ports = 1 in my code, but it was probably my fault. Removing it allows the chain to move forward, but when the dismantling country clicks the option A to take Nyasaland, it fails to be awarded.

In 96096, random_country requires that the satellite have a port:

https://github.com/moretrim/ccHFM/blob/47becfb5ccb5c1e4786bb500f718b524df991b20/ccHFM/events/GreatWar_Events.txt#L11440-L11452

Removing that requirement allows Nyasaland to be distributed to the dismantling country, but its provinces are blockaded. For this reason, this could be intentional. My general preference is to have the colony be distributed, and then the receiving country can decide what to do with it later.

Progress so far:

Portugal dismantlement now awards all colonies that have cores.
UK dismantlement now completes before time is up, and will award Pakistan, Nyasaland, Ceylon, and Burundi, in addition to all other colonies that work through the mechanism of releasing a puppet from the dismantled country.

To investigate

Awarding colonies and overseas states that otherwise do not function, such as the Suez Canal, Pacific islands, and Caribbean islands.

Well, enabling awarding of overseas states and colonies without cores was easy enough, uncommenting code, but some issues remain with that code. Some others are referenced in #16, and seem like reasons why the code was commented out.

Additionally:

Ulster releases to the war victor instead of to Ireland. This is fairly easily fixed.
Arguably it does not make sense to not award Moluccas along with the rest of Indonesia in The Return of Our Colonies.
In dismantling Russia, Kazakhstan, Yakutia, Astrakhan, Dagestan, Chechnya ,Circassia, and Kalmykia tend to scramble because they do not release as intended. They instead release through the generic process that releases unciv colonies. It is debatable whether they should release as independent countries or as satellites to be transferred.
Something is breaking the random releases in Russia dismantlement. Currently investigating, but it seems to be in the first Ural Republic release, which is itself a random release (there is a second random chance of it releasing as a non-ally). The problem seems to be with any_core = { owned_by = THIS }. I have also tried replacing THIS with RUS. I am hoping--and proceeding with the idea--that this part of the code is not needed.

So what remains to me, is to work out these and the issues stated in #16.

Addressing topics from #16:

the coin flip to hand out a colonial state will be used on colonies with cores, but is not allowed to give away core lands and this can lead to a pointless loop of notifications and infamy (affects places like Russian Manchuria)

This seems to be partially fixed already. However, I suggest that the possibility of having to pay 2 infamy multiple times through repetition of event 96011 (as the player) is the most noticeable remaining problem. The current version of this problem regards the coin flip alone, since it seems likely that Russian Manchuria is no longer an issue--or at least I could not replicate an issue with Russian Manchuria anymore. I suggest that it is not necessary to make the player pay either 1 or 2 infamy for what is currently at most a state such as the Suez Canal, and is often instead the Caroline Islands or a set of Caribbean islands.

When compared with the 0.05 infamy that one expects to pay for Comoros, or the 0.5 infamy for Cambodia, for instance, it does not seem balanced. Therefore, I suggest making the cost 0.05 and removing the middle option ("Our only interest is limiting..."). This value is of course subject to debate, but I would not recommend anything above 0.5. In tests after the other fixes already attempted, I am seeing the event repeat fewer times, and definitely not in an infinite loop.

Another alternative could be to change the entire system that the coin flip events use, but this seems to be a possible separate issue. For instance, the infamy could be applied upon awarding the state instead, although ideally this should include an option to say no.

possibly related to the above, the on_action pump event needs a review of its conditions: it restarts the dismantlement process as long as the dismantled country owns an overseas, non-core, or colonial province

I can no longer replicate this issue, and so I think it may have already been fixed before I tackled all this.

out of the three events which hand out piecemeal territory (96011, 96016, 96017), the first is the only one to hand it out at the province level—occasionally leading to split states

I have attempted to give each of these the same method of handing out territory.

released India and Indian countries may not be properly excluded from the catch-up Westernisation decision

I suggest that this could be a separate issue from what I am dealing with, or at least I think it is worth tackling entirely separately.

the code to release Estonia seems iffy Germany does not have first pick on UBD, which seems questionable

I am not sure of the exact nature of these issues, and so it is probably best if I leave them to others to tackle.

Where previously I was seeing results like this, after I had added civilized = no to the generic colony release code (because of Russian cores remaining on the provinces):

These now look more like a possible intended effect of Russian dismantlement, with the random releases working:

With the latest fixes, things are at a point where they need to be further tested. Given that the pre-fix dismantlement is acceptable if not perfect, I want to do some testing to make sure that game-breaking bugs are not being introduced.

Noticed details and issues:

Sometimes only West Java has Indonesian cores, which causes a multitude of Indonesian minor tags to be released as dismantlement puppets. This seems to be a consequence of organize_indonesia, which only applies cores to any_owned.

moretrim / ccHFM

Debugging errors in dismantlement #138