Process for Archiving *and Unarchiving* packages?

[ugroempi]

Dear CRAN Task View Editors (CTVEs ;-)),

thanks for the good work on moving the task views to github, I think that this will make maintenance quite a bit easier, at least for me (though making time remains a problem).

When checking on the archived packages now, I found several packages that were previously archived but are now back on CRAN, because maintainers eventually updated them (in my case, eight packages that were archived since Nov 19 are on CRAN again). Mostly, packages were either fixed within one month or within something around six months.

I suspect that the modifications to task views because of package archiving so far involve manual work by Achim @zeileis because suitable changes to the task view text are presumably hard to automate. Achim, I admire your apparently infinite amount of energy, but wouldn't it be better and more friendly for your time to automate something like

• an "archived" tag for a package in the task views
• an e-mail to package maintainers (that their package was/will be removed from task views and it is up to them to contact task view maintainers to get it included back in, once the fixed the package)
• and an e-mail (or an issue) to task view maintainers who can then handle the changes for the task view?

In the light of the fix times observed in my little sample, I would probably give it a few weeks (during which the package is flagged as archived in the task view), and would either remove the archived flag if the package comes back during this wait time or would eventually remove the package from the task view.

Best, Ulrike

[zeileis]

Thanks for the initiative, Ulrike! A couple of comments about this:

• As you note, it is hard to properly automate this because omitting or re-adding packages to task views often require changes in the text that are often (but not always) easy to do for a human but not completely straightforward to automate.
• My work on omitting archived packages is pretty easy and doesn't cost me much time. I receive an e-mail if a package from a task view is archived and I quickly adapt the task view file correspondingly. This is just a couple of minutes per week on average.
• Re-adding unarchived packages is much harder because the paragraph from which a sentence was deleted might look completely different by now. This is not so much a problem if packages reappear after a week but if it takes a year or so.
• The majority of packages that is archived on CRAN is not resurrected again. Some keep being developed elsewhere (e.g., on GitHub) but many of these projects are just dead.
• In many cases packages are archived because the maintainers did not respond. So sending them additional e-mail about what should be done for the task views will not improve the situation in such a case.
• The task view files on CRAN need to be updated quickly because otherwise links are broken and installation will throw error messages etc. This is why I do the adaptations of the task view files rather than informing the maintainers about it. I used to do that at some point but my life got a lot easier when I just did it myself.

Having said this: It would be possible to automatically change the links in a task view file when a package gets archived and to only send an e-mail when it was archived for more than, say, 4 weeks. But I'm not sure it is really worth the time and effort to do so. I'll discuss it with the CRAN team, though, to get their opinion about it. It wouldn't help with the majority of package archivals on CRAN, though, because most packages are not unarchived after 4 weeks (if at all).

[ugroempi]

Thanks for the quick reply! In my case, of the 20 packages that were archived since 2017, 10 got revived (one additional package is on and off). In recent years, archiving on CRAN has not only been due to unresponsiveness of maintainers in terms of answering e-mails, but also due to revision requests with ambitious deadlines that maintainers can't always keep (and some maintainers maintain packages for which they do not have full grasp of all aspects of the code, which makes it even harder); in such cases, the fix is made once the package maintainer finds time. I believe that this problem is on the rise, because CRAN has become more aggressive in terms of archiving packages for relatively minor deviations from policy.

The following comment is conditional on that it does not cause too much work, I have no idea how hard it is to automate something like this: I personally think that it would be good to just change or even remove the link and add "(archived)" for an archived package, in order to avoid error messages and make the status transparent to readers. I would opt for having a standardized automated e-mail sent at the same time. Individual task view maintainers could then decide how they want to handle such cases; I personally would go for waiting a few weeks, but others might make different choices.

Let's see what the CRAN team think.

[zeileis]

Re: CRAN archivals. I'm not here to defend the CRAN team. I'm just saying that their end of the problem is also not easy. They check almost 20,000 packages and have to deal with the corresponding maintainers. If in a small "positive selection" half of the packages are not revived, you can imagine how much work and how many non-response (or partially responsive) maintainers there are overall.

Re: Amount of work. Writing the code for what you propose will take me more time than handling all package archivals for all task views in a year. It would require a different workflow on my end and, more importantly, on CRAN as well. In case we send out e-mail, I would also expect that responding to follow-up e-mails will take more time than just editing the task view files at the moment. But as I said, I'll discuss it.

[ugroempi]

Re: CRAN archivals. I don't mean to criticise CRAN policy, which may very well be the best way to go, given the available person power, and CRAN are also approachable, if one contacts them for a reasonable extension of time.

[FlorianSchwendinger]

What about a simple approach, where the archived packages are added to a csv file (or sqlite) in the git repo, with name, archive date, and the git commit hash before the removal. So it would be possible to automatically create an issue containing the hash of the last version of the task view the package was listed if the package reappears.

[hwborchers]

Would it be possible to open an issue for every package that got archived, in each task view where it is mentioned? Then the maintainers have a hook and can discuss what to do, e.g. write to the author or simply forget it. My experience also is that (for the Optimization and NumericalMathematics Task Views) more than half of the archived packages do reappear.

[zeileis]

OK, I had some discussions with the CRAN team and a programming session with Kurs and I think we have a good solution now. You can try things out when you install the latest development version of the package from CRAN:

install.packages("ctv", repos="http://R-Forge.R-project.org")

When processing task views for CRAN we now obtain the list of archived CRAN packages and handle them by keeping the links in the info text (with comment: (archived)) but we exclude the packages from automatic installation. To see this in action, try

ctv2html("MyView.md", cran = TRUE)

where you replace "MyView.md" with a task view name. Ideally including some archived packages. (Unavailable packages are also handled similarly.) Also

check_ctv_packages("MyView.md")

now distinguishes between packages that are not available at all (e.g., when misspelling a package name) and a package that is currently archived on CRAN.

When we switch to this workflow, there is no need anymore to immediately update the task view files. Instead we will wait for a certain period (current idea: 4 weeks) before we notify the corresponding task view maintainer (plan: via an issue). Hopefully, the package has been unarchived by then. If not, the task view maintainers can take action themselves. If the problem still persists after an extended period (current idea: 6 weeks), i.e., the package is still archived and the maintainers have not changed their task view file, I will remove the package from the task view file and close the issue.

I hope this will be a big improvement because there will be no need for action for a certain proportion of archivals. And for the others there will be closed issues on GitHub that can be revisited later on.

[ugroempi]

Thanks Achim, I just tried it, and it works great!

Two suggestions:

• I'm wondering whether the "unavailable" tag for mis-spelled (or non-CRAN) packages should not also come with an "unavailable" category at the bottom (like the new "archived" category), because this allows maintainers to spot such mis-spells somewhere in their task view in a central place.
• I personally would opt for skipping the step that you (or someone central) remove a package from a task view if the maintainer does not visibly react: Occasionally, there may be reasons why maintainers want to keep archived packages (in my case, I want to keep planor, as long as it works, and it would be nice to keep it with an official tag, rather than as a text string only). Also, the task view maintainer may be in contact with a package maintainer and expect the package to go live again soon. In any case, with the new system, it does not do any harm if an archived package remains in a view. If there are many archived packages, users of the view will see that the view may be a bit outdated, but making that visible is more good than bad, in my opinion.

[zeileis]

Thanks for checking, Ulrike, much appreciated! Regarding your suggestions:

• The maintainers should keep running check_ctv_packages(...) which gives you a much quicker way of detecting such problems than looking at the end of the package list. But I will also consider adding it.
• I'm not convinced that it is a good idea to keep archived packages in the task views. We really want to generate a bit of pressure to bring such packages back to CRAN or at least into an active GitHub repository or so. In the case of planor: Is there no one in the community who would adopt the package?

[ugroempi]

I agree that intentionally keeping archived packages in a view for a long time should be the exception. In case of planor, I believe that keeping the package in the task view does not decrease pressure (having more places that highlight the need to fix something is rather more than less pressure, as far as I see it). There is a very busy maintainer who has no understanding of the more technical aspects of programming in the package, and the programmer is long gone. I think that keeping the package visible for the community in spite of its being archived makes sense. I might contact Hervé, whether he would be willing to hand over maintenance to someone else; if yes, I might try to help him to find someone.

[zeileis]

Re: planor. If he doesn't want to hand over maintenance, it might still be a good idea to put the package on a public repository (e.g., GitHub) and let others help with fixing the problems in the package. Then you could also link the package with github(...) in the task view until it reappears on CRAN.

[ugroempi]

Well, yes, but given his time budget and his (assumed) lack of experience with github, I doubt that this is going to happen.

[zeileis]

OK, I've released the new ctv version on CRAN now: https://CRAN.R-project.org/package=ctv

Based on this we will roll out the new workflow, hopefully in the next days. And then we can reassess later this year which improvements we can make in the workflow, specifically (but not only) with respect to handling archived packages.

Closing this issue now.