Work with the *actual* professionals at opening up code

[pdehaye]

Academic code has long been criticized for its quality.

So much so that a new profession had to be created, called in the UK "Research Software Engineer".

These jobs are often filled by people who work for the private sector, or the public sector working on very large projects where professional code is a requirement for the infrastructure (like NASA, CERN or other large scientific collaborations), who do enjoy working with scientists, and progressively manage to get funds or help attract funds to work on smaller projects.

In the UK, there is a professional society of those engineers. Were they at any point consulted before releasing this code? How was the decision made to get help from the private sector instead?

(for information, there are a lot of CERN people who were for the past x weeks stuck at home)

[phelps-sg]

This issue does not seem very constructive. The code has finally been released as open-source (better late than never) and there now seems to be some attempt to build up a community. Professionals can contribute to open-source projects. Professionals can open specific issues. Professionals can submit pull requests with improvements. If there are specific concerns with the quality of the software, the most constructive approach would be to document and resolve them. If there are specific concerns with the QA process, then again these can be documented, and professionals can volunteer to e.g. perform code-reviews.

[omalaspinas]

I think on the contrary that the decision process of what happened is key here.

Saying "we are trying to make it better" is not good enough anymore (although it's a good first step), and let's hope the community will make our code better. Although community can help with the future it cannot undo what has been done.

The damage made by this publication is HUGE. I am already starting to read people making a parallels between this disaster (yes this is the correct word) and climate change modeling. I guess you can infer what the conclusion of a D. Trump will be now on the problems with modeling and simulations.

I want to point out that I am not attacking any particular individual here. I think everybody is doing their best to try to save what can be. I do not think anybody acted in bad faith neither at any point. I do not think either that we should start pointing fingers. But how things developed is important for the future of this project. And contacting people at privately funded companies is not sustainable approach, especially in publicly funded academia projects.

The whole scientific community cannot continue to perpetuate these "short term projects" where everybody contributes for 3 months (years?) and then it's over. We need something more robust and long term. How many academic codes have a truck (bus) factor of 1-1.5 (including mine)? Or even worse, like 0.5, because the original developer disappeared but the code is still used? This is all about the politics of science and needs to change.

The thinking goes of course far beyond this particular example but providing answers to the questions bout the different choices made is key for the future or all of this will be for nothing.

[pdehaye]

My comment is maybe destructive of the mindset in which this code has been shared on GitHub. This would be a good thing.

It hasn't just been shared on GitHub so the professional community can improve it (which is a good outcome that should be encouraged). It has been shared publicly because it is good professional practice in science to do so, for a myriad of reasons beyond improving the code.

Another key aspect, for instance, is preserving the integrity of the scientific process, or actually optimizing the level of trust the general public has in the scientific process (blind trust in the scientific process would also be bad, as we will undoubtedly see very soon discussed publicly).

My original comment was precisely meant to focus on the process that lead to these professional epidemiologists seeking help from professional software engineers.

I support @omalaspinas' phrasing of around "bus factors". The following example is not meant to be personal, but could very much inform the discussion. In my original questions, I am curious particularly if there is anything specific in the epidemiology community that has led to the current situation (maybe they are exceptional in being a science that is clearly progressing through crush periods during crises, at each proto- or full blown pandemic; the fact that professional epidemiologists have anticipatively built a SAGE subcommittee seems actually the exception, but they themselves admit it was geared exclusively at influenza because that's what they got funding for -- i.e. a recurrent threat). We have a situation for instance, which has led us to having a very small group of people, probably a handful, capable of running this code at some point. One (if not more of those people) has since had to go into self-imposed quarantine, which he has himself broken against government advice. Regardless of the reasons, this has led in turn to a targeted information campaign against him, which got him to resign from the SAGE committee he was working on. This is not quite a bus factor of 1, but it is clearly crystallizing a huge amount of systemic risk into one person.

[weshinsley]

Some might consider Microsoft and Github to be actual professionals; as I see it, their offer for help included the capacity, expertise, and relevant modelling experience to review the code and make it more reviewable in a tight timeframe, without requiring us to top-manage or train individual coders - none of which we had/have any capacity for.

[phelps-sg]

OK, but I don't think involving professional software-engineers would have necessarily made things any better. I've worked as a software-engineer in both the private sector and academia. There is enormous variability in the quality of software projects in both domains. For example, in the private-sector I have seen shockingly-bad code deployed into production to manage national infrastructure, which fell apart as soon as it was released. Meanwhile academic programmers have given us BSD Unix, one of the ancestors of MacOS. If there are any lessons on the decision process on the software side I think they are about transparency, about the importance of making code publicaly-available and open-source right from start, so that it can be scrutinized, used and improved-on by a wide community of scientists and practitioners. On the scientific side, we need to ensure that all science used in policy-making is thoroughly peer reviewed before it used to inform critical decisions. This means accelerating the peer-review process, bypassing prestige publications in favour of those that can provide rapid peer-review turnaround, elicit and publish ongoing community-discussion of the papers, and open-access publishing. The target-culture in UK academia needs to be quashed. UK academics are highly pressured into demonstrating the that their research has high socio-economic practical impact, which can encourage a self-publicising unethical culture of promoting one's own methodology and work without sufficient discussion of the caveats. This was one of the reasons I left academic life. The behavioural insights team is a good example of this; these are academics who formed a ltd company to provide consultancy around their work. I do not believe that these scientists are in a position to give impartial government advice with a full discussions of the caveats of their recommendations. Anyhow, I don't think pointing the finger at academic programmers is insightful or fruitful for learning lessons here. Moreover, the 'Issues' section of a github repo should be reserved for constructive discussions on how to improve the code in the repo. If people want to have a discussion about software-engineering quality in academia or the wider caveats of this type of modelling and its use in policy, then there are more appropriate forums.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, 11 May 2020 10:37, Orestis notifications@github.com wrote:

I think on the contrary that the decision process of what happened is key here.

Saying "we are trying to make it better" is not good enough anymore (although it's a good first step), and let's hope the community will make our code better. Although community can help with the future it cannot undo what has been done.

The damage made by this publication is HUGE. I am already starting to read people making a parallels between this disaster (yes this is the correct word) and climate change modeling. I guess you can infer what the conclusion of a D. Trump will be now on the problems with modeling and simulations.

I want to point out that I am not attacking any particular individual here. I think everybody is doing their best to try to save what can be. I do not think anybody acted in bad faith neither at any point. I do not think either that we should start pointing fingers. But how things developed is important for the future of this project. And contacting people at privately funded companies is not sustainable approach, especially in publicly funded academia projects.

The whole scientific community cannot continue to perpetuate these "short term projects" where everybody contributes for 3 months (years?) and then it's over. We need something more robust and long term. How many academic codes have a truck (bus) factor of 1-1.5 (including mine)? Or even worse, like 0.5, because the original developer disappeared but the code is still used? This is all about the politics of science and needs to change.

The thinking goes of course far beyond this particular example but providing answers to the questions bout the different choices made is key for the future or all of this will be for nothing.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[pdehaye]

Would you have recommendation for such a forum, and how it might avoid the obvious retorts that the critique coming from there would be self-interested? (as well as hindering progress against the virus)

[phelps-sg]

The model is being widely discussed in the media (both mainstream and social) and there are some very insightful articles. See e.g. Nissam Taleb's critique of UK Covid19 scientific policy, which I personally think is spot-on: https://www.theguardian.com/commentisfree/2020/mar/25/uk-coronavirus-policy-scientific-dominic-cummings Also, Twitter is a good place if people want to let of steam and rant. Do not get me wrong, I do think that specific scientific and technical issues with this model should be documented transparently and the Issues section on the repo is a good place for this, and I would definitely encourage people to document specific shortcomings of the model and software. However, it is not useful to rant here- there is no social benefit to decreasing the signal/noise ratio any further.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, 11 May 2020 14:31, pdehaye notifications@github.com wrote:

Would you have recommendation for such a forum, and how it might avoid the obvious retorts that the critique coming from there would be self-interested? (as well as hindering progress against the virus)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[Feynstein]

I am agreeing with those who say non-technical criticism of this model in here is not a good idea. The politics around this model need to be taken elsewhere. There is also the issue of this code not being reviewed by professionals in software development. There is a lot of these kind of codes that exist everywhere. It's not just about epidemiology, it's also in a lot of engineering field like fluid dynamics and applied physics and I think that in order for this code to be criticized fairly all of these kind of programs must be viewed equally, this is really what is not going on atm. I also think that funding is the most dominant problem with these kinds of code, as scientists don't have the funds to hire a professional to make their code better. I think I am such a professional and that's exactly why I came here to help. Instead of ranting you're kind of professionals (computer science only) should also fork this code and start refactoring it in a more object oriented style than it is right now. It will make it easier to test and to add new features. As right now only those who knows how it works can change it without breaking it.

[Feynstein]

The academia code mindset needs to be changed and it needs more funding for verification and production-readyness. But this is true for EVERY academic code, so unless you can check with every western lab in the world if they have something like that, that is being used for critical real life problems, you need to stop ranting and start to work. Some of these science codes are being used in very critical parts of our society, I talked about fluid dynamics, dams and pipelines are the best example. Nuclear power plants also... Then biology, space, chemistry, aerospace, etc, etc. So basically... You just can't check them all atm and the more pressing issue that is concerned for this particular repo is to start working on it or shut up...

[pdehaye]

I think the more pressing issue is in whether there is anything worth salvaging here.

[Feynstein]

As I've already stated... When I was an undergraduate in physics, most of my friends summer internship was to translate this kind of code from Matlab or Fortan to C++. But they don't have good experience with coding and it ends up often in the exact state in which it came in but in another language. Why scientist choose these undergrads instead of professionals? Money.

[omalaspinas]

1. This not about ranting it is about trust. The public and developers (not the CS but more generally people contributing) need to know why this mess happened and what steps might be taken to avoid any other problem. There is a history to this project that other open source projects might not have.
2. I am no computer scientist neither is @pdehaye. I have been developing open source simulation libraries for about 15 years now (fluid dynamics in particular). Patronizing people because they are "just" CS people will certainly not help.
3. I am not comparing with other scientific software. But this one has a particular importance. It was used in real life which makes the transparency process even more important.

My aim here is certainly NOT to point fingers at people as I said in #179 and say what went wrong from the technical point of view with this project. But this is hurting all academia and the modeling community in particular that you are being so dismissive of this kind of criticism. Making the software work in the end would be great, but there are greater issues to reform trust.

[Feynstein]

@pdehaye you can't tell if you haven't tried. I made very good leaps of improvement over the weekend concerning data transport and loading. When I will be in the thick of it I'll be able to say. You can't rigourously say that if you only looked at it.

[Feynstein]

@omalaspinas I agree, but you can't overlook the fact that you are criticizing this code when you know many more examples exist in very critical fields. It's unfair not to look at the issue in general. And it can't be done here.

[Feynstein]

Try to find a git repo of another code and get in there and criticize it. Then we'll be able to speak fairly. Come on, do it, I really want you to do it. If you don't want it pretty much proves my point.

[pdehaye]

This is not an issue of fairness towards developers of the code, but fairness towards those who have or will suffer from the fact that political decisions were informed by this code or will be.

Your attitude might be compounding on this problem.

[Feynstein]

I don't have any attitude about it, you know it's true. I think that some of these codes have failed in the past and it was never this politically charged. It might have had way larger repercussions than everyone knew. The fact that this particular code is in the spotlight doesnt change a thing to the other codes. I know for a fact that a pipeline failing or a oil spill or a dam cutting power to thousands might have been caused by such kind of code but it was never publicised. Politics dont have their place on github and you need to ask for change in the global scheme of these codes, like about funding. But yes I think that if we are to use the scientific method to deal with this we need to look at all those codes and not only this one, there are pobably at least as much victims from failures of other codes, trainwrecks, and everything. If we dont look into all of them it becomes politics and it needs to be dealt with a political mindset.

I also want to apologize a bit I just woke up tired and there is a language barrier between you and me this is why I might come out too strong or with an attitude. It's really not intentional, I try to use the words that I know best and they might not be the most appropriate for this discussion.

[Feynstein]

You know what... I'll try to find a scientific spaguetti code and I'll tag you in one... It will help people understand how many ramifications these issues have and it will lead to the whole concept of scientific code to be reviewed. I also saw your background about mathematics. It seems to me that the project you are working on was specifically designed for code development. I also know that mathematics have a very close affinity to computer science. In my univerisity there is actually a double major program in both of those fields. It would be expected from a mathematician point of view to think that other codes should have had the same opportunities. It's not the case obv.

[Feynstein]

And finally... I want you to help contact the CERN people in order for them to work on this. Like be constructive and give me their references and I'll be sending emails, I probably know some of them anyway. Saying these people have nothing to do doesn't help and is not constructive if you cant do anything about it. Oh yes, I'll be referencing your first comment obviously to see what they think of it.

[Feynstein]

I want you to look at these twitter threads also about my first outburst of a comment #179 in here. Real professionals also agree with me. Like a senior principal engineer at Dell. https://twitter.com/HpcJohn/status/1259244324795056130?s=20 https://twitter.com/HpcJohn/status/1259245725654933504 https://twitter.com/GraziosiSergio/status/1259395123290439680?s=20 https://twitter.com/o_guest/status/1259246180913106949?s=20 https://twitter.com/johnrove3/status/1259242280252248075?s=20

[omalaspinas]

Maybe we can talk in French ;-) Apart from our weird accents I think we will understand each other.

The important part for me is not that "other codes are also bad so leave this one be". Bad luck for this one it became open after 15 years and people are playing hack a mole to correct the bugs (and it's good). I'm really sure that there is a lot of competent people here trying to salvage what can be from the code and the underlying model, and I'm blaming no one doing so. But you can't ignore part of the history here and the spotlight that this particular model has been under.

There are steps that IMHO must be taken to restore the trust that has been lost. And this goes through a disclosure of what happened with this project.

This particular code was, is still partly used, and from what I read in the issues will be used as a basis for a lot more than just this covid epidemic. It is planned to be used fo

edit: missing word.

[Feynstein]

@omalaspinas I strongly agree with eveything you said. Steps need to be taken in order to restore trust. The past is the past and we need to look forward with this issue. I think that from now on the best way to restore trust if for people to

1. Run it as is and replicate(or not) the results given publicly
2. Improve the model in order to make it workable and more efficent and more replicable.
3. Compare and minimize the results error with the current data.
4. Regenerate all the predictions

[Feynstein]

Even if we prove the first code is wrong it will be a good thing because it will show that the whole scientific process is being respected here. French canadian saying: "Faute avouée, à moitié pardonnée."

[pdehaye]

And finally... I want you to help contact the CERN people in order for them to work on this. Like be constructive and give me their references and I'll be sending emails, I probably know some of them anyway. Saying these people have nothing to do doesn't help and is not constructive if you cant do anything about it. Oh yes, I'll be referencing your first comment obviously to see what they think of it.

It's too late now. Life has restarted essentially in Geneva. Schools opened today. There was a moment in time when people wanted to help, when there were hackathons of tens of thousands of people - some organized by people in CERN - producing projects that went to the dustbins already. That was the ideal time to say "please help".

[phelps-sg]

@Orestis, the title of this issue is "Work with the actual professionals" which implies that "this mess" was caused specifically because Imperial did not work with "actual professional" engineers (whatever that means). This claim is highly debatable, and is not straightforward to verify objectively. There have been lots of software disasters perpetuated by "professional" software engineers (https://www-users.math.umn.edu/~arnold/disasters/). Meanwhile, students and academics in Universities have produced code that is widely, and successfully* used in mission-critical infrastructure, see e.g. Apache Spark. Was Linus of Linux fame an "actual" professional? No, he was a student in his bedroom when he released the first Linux kernel.

At some point there will be public inquiry into the whole Government Covid-19 response, and perhaps the committee's terms of reference will look into the software QA process at Imperial. Meanwhile, however, there is simply no point in submitting github issues with a title like this one. If an issue cannot be resolved, what is the point in submitting it on github? If Imperial were to suddenly hire or work with "actual professionals", and then _close_thisissue, would that really help the public's trust, increase transparency, or make the world a better place in any way?

There is plenty of other software on github that is widely used in real-life, and is not written by "actual professionals". Should we go around and submit an issue "Work with the actual professionals" on every such github repo? Will that help anyone? Will that increase transparency? Will the public place more trust in software as a result? No, it would just turn the whole open-source software community against you.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ On Monday, 11 May 2020 15:15, Orestis notifications@github.com wrote:

• This not about ranting it is about trust. The public and developers (not the CS but more generally people contributing) need to know why this mess happened and what steps might be taken to avoid any other problem. There is a history to this project that other open source projects might not have.
• I am no computer scientist neither is @pdehaye. I have been developing open source simulation libraries for about 15 years now (fluid dynamics in particular). Patronizing people because they are "just" CS people will certainly not help.
• I am not comparing with other scientific software. But this one has a particular importance. It was used in real life which makes the transparency process even more important.

My aim here is certainly NOT to point fingers at people as I said in #179 and say what went wrong from the technical point of view with this project. But this is hurting all academia and the modeling community in particular that you are being so dismissive of this kind of criticism. Making the software work in the end would be great, but there are greater issues to reform trust.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[omalaspinas]

I think the main criticism behind this provocative title is why hire a privately owned company to publish the source code (as made clear by @pdehaye clarification in the 4th post). Why not use the available teams in the public sector to do this work and have a sustainable way of continuing it? This keeps some kind of not so sane "secret" around the code that does not sound good in my opinion. We need sustainability around this issue not a patchwork of short term solutions.

[Feynstein]

@omalaspinas You actually just answered the concern here. This github repo doesnt have anything to do with the public decisions made around the algorithm. Hence, such a title is very purposefully non-constructive and it annoys me to a point I really want to be vulgar. I won't. If you want them to use the professional public sector teams, just write an email to the leader behind this repo and ask him yourself. If this idea hasnt been already thrown around I'm sure they will make good use of it. Even if you don't have an answer. And also, maybe they are working with public teams, you simply can't tell... Its maybe not public... This kind of issue needs to stop on this repo, its toxic. If you have any inquieries about this thing being used by your government, you wont make anything change in here. Do you understand that?

[Feynstein]

No government official will come look at some random issue on a github repo, they will ask the opinions of experts in this field, like Neil Fergusson. The debate about if he's competent enough is a completely other thing. If you think he isnt contact other renowned epidemiologists and ask them to work on modelling, or their predictions. Then contact your local deputy to make things change. Thats how real life works... When you're unhappy about government decisions, the real way to do something about is certainly not github.... I rest my case.

[Feynstein]

Everything you say is very constructive... we know ALL of it... but making things change in a democracy involves democratiquely elected people... contact yours to make a change or start a petition in the parliament... we can do it in Canada, theres one going around right now because some are angry about the assault style weapon ban in canada. Thats EXACTLY how to proceed.

[insidedctm]

The important part for me is not that "other codes are also bad so leave this one be". Bad luck for this one it became open after 15 years and people are playing hack a mole to correct the bugs (and it's good). I'm really sure that there is a lot of competent people here trying to salvage what can be from the code and the underlying model, and I'm blaming no one doing so.

There's a lot of implications in this entire thread that somehow different political decisions would have been made if only there weren't "bugs" in this code. Does anyone have something specific to back that up?

[dimpase]

You know what... I'll try to find a scientific spaguetti code and I'll tag you in one

we with @pdehaye know very good examples of scientific spaghetti code from our work on sagemath, which, among other things, took over a number of abandoned/orphaned libraries and kept them alive.

[Feynstein]

@dimpase then why don't you do anything about this one? Why do you say that even? It doesn't have anything to do with what I'm saying because it's only one example that you point out. There are many more out there. You guys need to stop being non constructive. That's the only issue here wtf? Why don't you contact those people and tell them to come work on this with us. Or look at the code and write relevant issues. What are you doing, showing off like that instead of being relevant.

[Feynstein]

Your arguments are mostly true, and I could add to them easily. I choose not to, because it's a complete waste of time.

[omalaspinas]

@dimpase then why don't you do anything about this one? Why do you say that even? It doesn't have anything to do with what I'm saying because it's only one example that you point out. There are many more out there. You guys need to stop being non constructive. That's the only issue here wtf? Why don't you contact those people and tell them to come work on this with us. Or look at the code and write relevant issues. What are you doing, showing off like that instead of being relevant.

Well as I wrote in #179 I actually contacted Neil Ferguson in March by e-mail . Still waiting for his answer. I guess that it will never come (I'm probably not important enough to be given any consideration). Now that all that happened you want me to contribute and try to save what can't be saved anymore? Not sure what you ask is fair... How do I know what will happen to my contributions? How can I trust that this code will be better handled than the previous one? I'm not saying that anything bad will certainly happen. It's just about trust you build with people. When you fail break the trust people had in you in the past, you cannot pretend nothing happened. At the very least some history/clarifications on the state of the project, how it started, what is its purpose, what steps were taken, etc must be added in the README. And this discussion certainly belongs in the ISSUES category of this project.

[zebmason]

Quite an amusing thread. I finally found this repo the middle of last week. Reviewed the code and thought oh dear, subsequently read The Spectator article which made me rather bemused when I heard that MS had worked on refactoring the code. I suppose I've got a long history in code maintenance on academic and blue chip code whereas those people are probably more used to greenfield development. Being something of a stoic I decided to see whether I could get some code changes pushed through. I've had one approved that was just 3 lines and am waiting to see whether something more substantial makes it. Must admit that I really should be looking for some paid consultancy but my usual customers are in aerospace and automotive...

[pdehaye]

There's a lot of implications in this entire thread that somehow different political decisions would have been made if only there weren't "bugs" in this code. Does anyone have something specific to back that up?

I am pretty sure uncertainties would have been communicated differently. Note that uncertainties could have gone both ways. It's hard to overstate how influential this model has been, see here in Le Monde for a detailed presentation of how the results were communicated to Macron.

And again, this code might still be used to influence policy.

[pdehaye]

I think the main criticism behind this provocative title is why hire a privately owned company to publish the source code (as made clear by @pdehaye clarification in the 4th post). Why not use the available teams in the public sector to do this work and have a sustainable way of continuing it?

Indeed. I updated the title to reflect this.

[pdehaye]

For the reasons on why it might be important to open an issue, even if it is not welcome, see here.

[Martin-Spamer]

Academic code has long been criticized for its quality.

This is not at all helpful, it is a rant, it is neither specific or constructive criticism.

So much so that a new profession had to be created, called in the UK "Research Software Engineer".

Then take a constructive approach, every skilled Software Engineer should know how to raise a defect; set out the expectation, set out the actual, highlight the divergence; and how to perform a code review, make specific suggestions, ask open questions, do not engage in what amounts to an ad-hominem attack that could readily be considered trolling.

There are undoubtedly issues here, any non-trivial software project will have issue, those are best addressed with professional indifference to politics or rivalry.

The OP should close this as an error of judgement, and raise specific issues in detail, not dig deeper.

[dimpase]

This issue is related to the refusal of the original authors to release the original (pre-2020/04/01) code, as well as attempts to hide bugs (?) in the original code, such as an interesting typo in the RNG code, cf. https://github.com/mrc-ide/covid-sim/commit/7902c37f69b52cd7c8b72f4e1f84f2e6676d0661 which might be pretty much invalidating computations done by this code before this fix (or maybe not, I don't know).

Has anyone performed e.g. the comparison of the RNG performances before and after the latter fix?

[insidedctm]

an interesting typo in the RNG code, cf. 7902c37 which might be pretty much invalidating computations done by this code before this fix (or maybe not, I don't know).

So have you tried setting it back to the original value and seeing if it substantially changes anything?

[dimpase]

The right thing, from software engineering point of view, is to make a specific test for the RNG, which merely tests that it produces what is claimed in the paper it is taken from, that the distribution it generates is not skewed.

On Tue, 12 May 2020, 14:57 Robin East, notifications@github.com wrote:

an interesting typo in the RNG code, cf. 7902c37 https://github.com/mrc-ide/covid-sim/commit/7902c37f69b52cd7c8b72f4e1f84f2e6676d0661 which might be pretty much invalidating computations done by this code before this fix (or maybe not, I don't know).

So have you tried setting it back to the original value and seeing if it substantially changes anything?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mrc-ide/covid-sim/issues/209#issuecomment-627362523, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJXYHACIQVZ6UNR3TZPIBDRRFIUVANCNFSM4M5WAYEQ .

[phelps-sg]

Has the original pre-2020/4/01 code now been released, or is this still outstanding? Is there an open issue specifically requesting this code?

This issue is related to the refusal of the original authors to release the original (pre-2020/04/01) code, as well as attempts to hide bugs (?) i

[Feynstein]

@phelps-sg yes #179 its a pretty long and politically charged thread though...