Create python script to convert downloaded implementation schedules from web app to new FoPITY python format

[mkmahajan]

The web app currently allows for downloads of the .cin file and a .csv file with the policy implementation schedule that's compatible with the old FoPITY excel files. However, we recently moved to using a python script for FoPITY, which uses a different format. That means it's no longer possible to simply copy/paste the downloaded .csv into a FoPITY file to recreate a partner's shared scenario.

Todd has suggested the easiest solution is to write another Python script that reads the .csv file and emits a Python script using the new format.

[jrissman]

Well, we now have a single Python script that handles all policy schedules (up to 9). The downloaded one from the web app only contains info for the active scenario in the web app. So we can't just convert it to a different format. We'd need to have a way to smoothly edit an existing Python script to add the new schedule data using the lowest not-yet-in-use schedule number. To avoid adding unnecessary lines, it would need to detect when the line to be added is the same as the default schedule for a given policy and skip that line.

It is something we could do, but I'm tempted to give this a very low priority (or not work on it at all), given that our partner could simply add lines to the Python script by hand, which is only needed for policies whose implementation schedules the partner customized. The web app makes the policies with edited schedules obvious by adding a 🕓 icon next to the setting of a policy with a customized schedule. Even if a partner uses a dozen or more policies in their scenario, they might only edit the implementation schedule for one or two of these policies. So, sometimes, it might amount to adding just one or two lines to the Python script, a task that could be done by hand in a matter of a few minutes. (There is a little more copying and pasting if it's a heavily subscripted policy, but this could be done quickly in a text editor with find-and-replace.) It might not make sense for us to do a large amount of work to build a computerized solution for something that might be relatively quick and easy for a partner to do by hand.

Any thoughts/data on how often you encounter partners that have customized the implementation schedule for dozens of policies in the web app and want that scenario moved to Vensim?

[robbieorvis]

It does happen with some frequency, because it’s much easier to develop policy scenarios in the webtool than in Vensim. I would make call it medium priority.

However, it seems to reason that if you still have an old copy of the FoPITY .xlsx file, you could drop it in and the run the converter, right?

[mkmahajan]

That is true that we have the FoPITY converter tool that Jeff made, but we aren't actively maintaining versions of the FoPITY excel files past version 3.3. So as we add new policies to the model, we don't have a correctly formatted list to drop into the first tab of Jeff's converter tool. But perhaps doing the work to maintain that would be easier than creating an entirely new python helper script.

For reference, I needed to manually convert WRI Mexico's policy implementation schedule from a downloaded file last week. Oftentimes our partners have introduced things we want to fix (such as customizing a schedule for a policy that they didn't end up implementing, or phasing in values starting from 2020 instead of 2022). Because of all the customization they did, plus looking out for these fixes, it took me about an hour to manually convert one policy schedule. But of course, I would have had to go back and manually make some of these fixes even if I'd been able to download their file and copy/paste it into FoPITY.

[jrissman]

Well, that converter tool was quickly hacked together because it wasn't intended for ongoing use. Using it involved a number of manual steps, which could be prone to someone forgetting something and introducing a bug. In the end, I don't think I even updated it to handle the final format now used in the Python scripts, which avoids duplicating lines and puts the default schedule at the top, so there are some steps the tool doesn't even handle. Plus, as Megan noted, we'd need an up-to-date FoPITY in the old Excel format to convert from, and we don't maintain that anymore, because maintaining it in the old format would defeat the purpose of having switched to a new, easier-to-maintain format.

If you really need the web app to be able to export schedules to Vensim, I think Todd's suggested approach of using a Python script to write a new Python script is the right one. It's just more complicated than Todd realized because the Python script has to read both the existing FoPITY python script and the web app's downloaded CSV file, then make a new Python script that combines the two. I do think that's best from a software architecture standpoint.

Megan, how do you feel about Python programming? Is creating this new Python script something you might be interested in attempting on your own?

[mkmahajan]

I don't have experience with Python programming, although taking some intro courses and getting some practice were actually on my OKRs for this year. I could note this as a back burner project once I have some time to get up to speed on Python.

[robbieorvis]

@oashmoore didn't you make this?