April sitrep

[james-large]

Hello all, hope everyone's happy and healthy during these times. In general, I think we're all aware that world and life events put projects like this on the backburner. This is just a faux issue to get back up to speed a bit on the project

The general short/mid term goal was (and still is) to get a master push out and pypi package updated with all the basic, solid, and compatible functionality we've added/are adding. That way, the project serves as a base of good core functionality that we can just implement new algs/networks into, and update as needed.

Very broadly, things we still want in:

• Ease of installation: the setup is pretty solid I think. I had a play around with the docker pr #35 last week, and while I didn't get a gpu version working, I could use it with tensorflow-cpu (I believe anyone claiming to have successfully set up gpu usage for docker on a non-fresh windows installation without nesting a linux vm to run docker on (lol) is either a lying or a god). I'm happy that this is working in principle. Perhaps we do add the option to build for cpu only, either a separate script for a separate docker image, or an option into the same script/image.

  On the binder plugin, that is all working in principle too, but wont work on master until it's actually on there obviously

• Ease of use: Examples/documentation. Example notebooks are ongoing under #35 also. A job of going through the docstrings will be needed as one of the final steps before going dev -> master. The overall API is solid, however there's the backend ongoing issues and how we want to handle them #45.

  On the API though, base sktime is going through it's refactoring, where there'll likely be some knock on updates needed for sktime-dl. A word on this @mloning? Is the refactor leading into a master push for base sktime? What vague timescale do you image on this given all that's going on, weeks/months?

• Functionality: Few main points under here

  • Networks. As before, there's always going to be new networks from the literature to add, this is just on going this regardless of short term goals. There are individual networks that need the generalisation process #18 though, see #46
  • Tasks. Classification/regression in general are all good, and forecasting via reduction is fine as well, barring base sktime updates to this. Other tasks can wait, in my opinion.
  • Meta. Ensembling, may need updates to generalise across tasks. Tuning the same, but also networks may need updates to expose tunable parameters. Callbacks are exposed, but there's no nice wrapping code for it at the moment, unsure whether this would be needed either
  • Experimental/analysis. Any functionality wanted beyond that which is already in base sktime. Network and training visualisations seem most obvious, whether we wrap some of the process or just expose it for now a la callbacks. Have actual keras model saving/loading if I recall correctly, but this could be expanded further to incorporate more of the sktime-dl model wrapper as well, will look into it

• Maintenance: Core tests and coverage are ok, obviously can be improved. Test coverage for macos and windows on travis would be good too. Don't think these are a must for the next push, but are things to improve over time.

Anything major I've missed you think?

[mloning]

Hi all, regarding the refactoring of base sktime, I hope to be done with that at the end of May at the latest, it will lead to a new release (v0.4.0), here's a short summary of what it includes:

• complete refactoring of forecasting API
• reorganisation of classifier/regressor functionality, but basic API is stable
• updated build/CI/distribution for Python>=3.8 including 3.8 on macOS, Win and linux
• updated docs

PR: https://github.com/alan-turing-institute/sktime/pull/246

[james-large]

Nice, sounds good. We could aim to restore parity and be ready for a master push somewhere around the end of May or first weeks of June then. If we get a massive kick in the arse and get a lot done in the next week or two, we can split it into two parts, 0.2.0 = the big functionality update (0.1.0 is distant memory compared to current dev), 0.2.1 = restoring parity to next version of base sktime

[Withington]

Great, I’m glad we are back on it.

Ease of installation: Docker – agreed, the TensorFlow Docker image only supports GPU on Linux (as per their comment here https://www.tensorflow.org/install/gpu). I’ve updated the README with an instruction on how to run our Dockerfile in CPU only mode. #35

Functionality: Networks #46 adds MCDNN regression, leaving only MCC and TWIESN without regression. My vote would be to leave these on the todo list but go ahead with release 0.2.0 without them.

[james-large]

1) Yeah the Tensorflow-provided setups are for linux only and work fine, there were people around saying you can get it working on windows too though. Provided a nice distraction for a while until it turned into annoyance.

2) Agreed leave on the todo in my opinion. On a pragmatic level, they are not the strongest networks on average, from the point of view of a user just wanting to benchmark the strongest approaches on their data, MCNN and TWIESN liekly aren't the ones to be using anyway

[mloning]

So, there are two remaining PRs now because we can go for a new release.

Refactoring in sktime has also moved ahead and should be ready for a new release at the end of the month. Shall we make a separate release with sktime-dl before that or check integration with sktime's new release changes first?

[Withington]

We should probably wait for sktime release. Shall we create a release candidate branch soon, to avoid adding too much more to the release? Proposed set of branches:

• master (reflects sktime-dl 0.1.0)
• release (create this from dev after PR #55 and #57 are merged. Release candidate for version 0.2.0)
• dev (new features not targeted for 0.2.0)
• any other feature branches (for merging to dev)
• any bug fix branches (bug fixes to release branch)

[Withington]

Are there any issues that need to be addressed for v0.2.0? I have some time to work on it for the next few days.

[mloning]

I think we just need to wait for sktime now, but if you like, you could work on some of the easier open issues (e.g. consistent handling of random states #53 or investigating the sporadically failing forecasting tests #56)

[mloning]

So finally, sktime v0.4.0 is out. I have a few other things to look into first, but we should be able to make any final changes to ensure we're compatible and then publish a new release for sktime-dl.

[Withington]

good going @mloning ! 🥇

[mloning]

FYI: https://mcfly.readthedocs.io/en/latest/ deep learning for time series classification with automated hyper-parameter selection

[Withington]

Hi team, I hope you are all well. I'm keen to get dev promoted to master, especially since jmrichardson is starting to use dev. What needs to be done? Is it just a case of switching dev to use sktime v0.4.1 and testing? If so, I can do that.

[mloning]

Hi @Withington, I agree! Ensuring everything still works with sktime 0.4.1 will be the main thing. We should also be able to replace the forecasting wrapper with sktime's forecasting wrappers in sktime.forecasting.compose

[james-large]

Hi @mloning @Withington

Surviving to some extent, pardon the deja vu on this issue. Hope you're both well also

I've tidied one of the branches I started a while back and made the PR for that, as I type it looks like the linting failed, will fix in a sec.

@ABostrom did a PR (#59) that hopefully caught most of the changes for base sktime. If we trigger another run of the tests for sktime-dl dev and they pass, are we generally happy to take that as full compatibility? I believe we may want/need to update the minimum version to 0.4.0 or 0.4.1, it is currently >=0.3.0 (https://github.com/sktime/sktime-dl/blob/dev/setup.py#L53). Don't think 0.3.0 is compatible with -dl any more.

On versioning, tensorflow 2.3.0 came out a week or two ago. It seems to have not brought in any new conflicts and be used with -dl as is, miraculously. We could include another test env for this potentially, or change the 2.1 test to 2.3. 2.1 is where the changes to tensorflow-addons happened, so there may be merit in explicitly having 2.1 (https://github.com/sktime/sktime-dl/blob/dev/.travis.yml#L27).

In terms of additional functionality for 0.2.0, I'd be very happy to shelve too much more and prioritise the dev->master push. 0.2.0 is a completely different beast to 0.1.0 already, waiting further and adding an extra <5% difference seems unnecessary. There's always 0.2.1 or 0.3.

What should be done though, as @mloning said, is replacing the forecasting via regression tests.

I have intentions of implementing a wrapper for passing data in as a generator to get around a gpu memory issue I'm having with some data. This is something that would make using a lot of networks easier on non-tiny data for the user dipping their toes into dl and gpus without expensive hardware. I'd be happy leaving it out if we want to do a push asap (today/tomorrow).

[Withington]

Great to see that sktime-dl is moving on :-)

On "are we generally happy to take that as full compatibility" - I'll also run the notebook examples when I get on to this tomorrow.

Also, the forecasting test was removed in #59 so I'd like to understand where we are with that (see question raised in that PR). Is that the same as @mloning point on "replace the forecasting wrapper with sktime's forecasting wrappers"? Is anyone already looking at that?

It's worth keeping a test env for tensorflow 2.1 while users are likely to still be using it.

On your data generator work - I'm happy to leave that out of this release, in the interest of getting sktime-dl 0.2.0 past the finishing post.

[ABostrom]

Yeah. Deliberately left the forecaster tests out of that PR, as I couldn't find the analogous functionality from sktime anymore. But @mloning may know more etc.

[fkiraly]

obsolete planning issue from 2020