`sktime` time series anomalies/changepoint interface upgrade, moving `skchange` to 2nd or 1st party?

[fkiraly]

Great package!

I was pointed to this by one of your collaborators, and wanted to let you know that we are currently reworking the anomalies/changepoints interface - the API had some inconsistencies for a while and we are planning to move it to a more mature state over the next minor release cycles.

It would be great to pool ideas and perhaps work on this together!

@alex-jg3 (sktime core developer) is currently driving the rework.

Relevant issues:

• base class rework long thread: https://github.com/sktime/sktime/issues/3214
• wishlist for annotation/changepoint algorithms: https://github.com/sktime/sktime/issues/6481 - we are populating this at the moment
• handling post-0.30.0 interface changes: https://www.sktime.net/en/stable/changelog.html#core-interface-changes

Input and feedback on the interface designs are much appreciated! Criticism especially, as you are a "consumer" of the interface.

Given the current state of skchange, we could also help:

• transition skchange from 3rd party to a 2nd party library, with synced API checks, CI, and indexing of the algorithms through the sktime index, while retaining it as a separate library.
• transition to a 1st party, i.e., moving estimators to sktime proper with ownership and maintenance assigned to authors.

What do you think? FYI @alex-jg3

[Tveten]

Hey @fkiraly!

I have actually been meaning to contact you about the points you raise, so it's great that you found your way here on your own through my collaborator.

First, I'd be happy to help on the rework of the annotator/anomalies/changepoints/segmentation interface. What is the best way for me to contribute? Is it mainly by following and contributing to the issues you mention? And get started by following the instructions here https://www.sktime.net/en/latest/get_involved/contributing.html?

I've also been thinking about whether skchange is best suited as a separate library or as a 1st party library down the line (in case you were interested). I currently enjoy the freedom that comes with creating a separate package, especially since it's still at an experimental stage, but I see the advantages of full integreation, definitely. Let me think about it.

Meanwhile, could you elaborate what making skchange a 2nd party library means? What needs to be done in sktime and skchange to make that happen? I know you have adapted several separate packages, e.g. tsfresh, to the sktime framework. Is that an example of a 2nd party integration?

[fkiraly]

Thanks for the reply, @Tveten!

First, I'd be happy to help on the rework of the annotator/anomalies/changepoints/segmentation interface. What is the best way for me to contribute?

Normally, a looser contribution pattern such as via issues etc would be what I would recommend, but the annotation module is "special" in the sense that it has been in an experimental stage for a while and we are moving it to a maturing stage, consolidating interfaces, so it is earlier stage than other modules.

Given this and your dependencies, I think it might be best if we have regular touch points, with @alex-jg3 and possibly users of skchange.

Concretely, I suggest:

• join the sktime discord, these are our primary channels for async developer communication: https://discord.com/invite/54ACzaFsn7. There is a dedicated channel in "workstreams", but "dev-chat" is also frequently used.
• consider participating in synchronous meetings, I would suggest at least one to coordinate, and try to set upa tighter asynchronous pattern using discord.

For synchronous, we have sync meetings on Mondays 12 UTC (workstream tech meetings) and Fridays 13 UTC (meetups, content varies, presentations or tech planning). Serendipitously, there is also the upcoming roadmap planning meeting 2024-2025 on June 21, i.e., very soon! It might make sense for you to join and suggest priority roadmap items based on your use cases and user base.

On the discord, we can send quick messages to schedule if any other timings are needed.

[fkiraly]

I currently enjoy the freedom that comes with creating a separate package, especially since it's still at an experimental stage, but I see the advantages of full integreation, definitely. Let me think about it.

I would say it's up to you, if you feel you have the capacity to maintain. As you say, the trade-off is between maintenance burden and agility. In any model, we are happy to help. As said, the annotation module is less consolidated than others, so it would be important imo to coordinate, especially when it concerns framework architecture and roadmap.

Meanwhile, could you elaborate what making skchange a 2nd party library means? What needs to be done in sktime and skchange to make that happen? I know you have adapted several separate packages, e.g. tsfresh, to the sktime framework. Is that an example of a 2nd party integration?

Let me be a bit more precise on what I mean with the different models.

The below applies for packages usable via sktime API by other users, i.e., packages or algorithms that are meant to have their own users as opposed to being consumers/users themselves of sktime and algorithms therein.

• 1st party: contribution is directly to sktime and packages maintained by the sktime team. Governance and maintenance framework is the same, that of sktime. Code-wise, estimators are part of sktime repository, and subject to sktime CI.
• 2nd party: there is a separate package, e.g., skchange, under separate governance and maintenance, licenses and owners may differ. There is technical integration and active cooperation both ways, e.g., PR to each others' packages. Communication and development is integrated via interfaces, e.g., following dev announcements on discord. Code-wise, estimators live in a separate repository, and CI runs interface conformance tests. Recent examples are prophetverse https://github.com/felipeangelimvieira/prophetverse, which follows the sktime forecaster API, or tsbootstrap https://github.com/astrogilda/tsbootstrap, which follows the scikit-base API while introducing its own base class interface that sktime adapts to transformers.
  • later it can be decided to transition to 1st party (bilaterally), or 3rd party (unilaterally), e.g., if maintenance capacities change
• 3rd party: there is a separate package but no active collaboration. The sktime interface of the 3rd party package is maintained unilaterally, by sktime, and the 3rd party package typically does not make efforts to be interface compatible with sktime unified APIs. Examples of this would be the tsfresh package your mention, or statsmodels, because neither projects have a commitment to compatible interfaces, share development processes, or commit resources to the integration.
  • in-principle there is a variant of this where the 3rd party package provides sktime compatible interfaces without active collaboration. We know this happens a lot in closed code bases (where IP management is the obstacle to collaboration), but we have not seen this in open code bases. Code-wise this works by using the contract checking utilities.

[fkiraly]

There are also some typical transitions we've seen over years:

• 3rd -> 2nd. In a case where the sktime maintained interface is substantially more convenient or accessible (e.g., parameter choice, documentation), the 3rd party may choose to "adopt" the sktime interface and become its maintainer, while retaining a separate package. The typical case would be research grade software or academic software.
• 2nd -> 1st or 3rd -> 1st. The typical case is where maintenance for the software ceases, e.g., due to small maintainer team and career changes, or corporate projects getting cancelled. In this case a 2nd party maintainer may choose to move the package to sktime (repo or as small onboard lib), or sktime devs may choose to fork an abandoned package into sktime to ensure continued maintenance. An example of this is pykalman, you can read up on the saga here: https://github.com/sktime/sktime/issues/5414, this is a fork made by sktime; an example of collaborative transfer is vmdpy.
• 2nd -> 3rd. The typical case is a startup in the time series space that uses sktime as an initial platform to gain notoriety and a user base, then divests its development resources from maintaining the sktime interface in favour of its own package or software, which may be open or closed. Examples are interfaces to statsforecast and temporian, both run by companies who initially helped maintain an integration and no longer do.

[Tveten]

Thanks for the thorough reply!

I think a 2nd party solution sounds like a good solution for now, then we can see what happens down the line. So a few concrete steps towards this is:

• skchange already runs interface conformance tests, I believe: https://github.com/NorskRegnesentral/skchange/blob/main/skchange/tests/test_all_detectors.py. So maybe not much to do on the skchange part of the code?
• Make skchange adaptors in sktime.
• I contribute to the ongoing annotator discussions on Github and Discord.
• Conform skchange to the new annotator interface.

Something like that? I'm very open to additional suggestions!

[fkiraly]

@Tveten, makes absolute sense.

I would suggest as an additional point, what would be very useful is joining the discussion and/or active development on improvements to the 2nd party developer experience, together with @felipeangelimvieira (prophetverse).

This PR came out of that: https://github.com/sktime/sktime/pull/6588 so now you can use check_estimator, and pytest will run the individual tests separately.

I will open an issue with some of the current improvements around this, and also "indexing", i.e., discoverability via all_estimators and the estimator search GUI on the web: https://www.sktime.net/en/latest/estimator_overview.html

[Tveten]

@fkiraly Fantastic!

[fkiraly]

Here: https://github.com/sktime/sktime/issues/6639