Preparing for vty multi-platform release

jtdaugherty commented 1 year ago

This issue is for tracking the status of numerous tasks as part of the recent/ongoing work to support Windows in Vty. These tasks are now aimed at preparing for a public release. This is mostly for me to track my own work but also to provide a place where others can see what's planned and discuss it.

Planned release/announcement date: October 30, 2023

Related issues:

People involved: @jtdaugherty @chhackett @shrykewindgrace

Relevant repositories and branches:

vty: refactor/vty-unix-split branch
brick: refactor/vty-crossplatform branch
vty-unix repository
vty-windows repository
vty-crossplatform repository

Relevant pull requests:

https://github.com/jtdaugherty/brick/pull/483

Tasks, roughly in order:

General
- [x] Figure out where the vty demo applications (in demos and examples) should live (probably vty-unix)
- [x] Move the demo applications and get them building
- Rogue and EventEcho demos moved from vty to vty-crossplatform; build with cabal build -f demos.
- [x] Decide on one or more public forums to announce the change in Vty's packaging
vty
- [x] Get all (remaining) tests building
- [x] Get all (remaining) tests passing
- [x] Write changelog with details of the package split (including what changed, what moved to vty-unix, and what went away entirely and how to deal with that)
- [x] Easy-to-miss change: vty no longer implicitly loads the user config from disk. That is now an application responsibility (to call the library routine that does so).
- [x] Do full Haddock audit, looking for references to things that changed/moved/went away
- [x] Write a migration guidance document
- [x] Write a platform package implementer's guidance document
- Here: https://github.com/jtdaugherty/vty/blob/refactor/vty-unix-split/PLATFORM-HOWTO.md
- [x] Update the README describing how vty needs to be used now
- [x] Set version to 6.0
- [x] Merge to master
vty-unix
- [x] Get all (remaining) tests building
- [x] Get all (remaining) tests passing
- [x] Do full Haddock audit, looking for references to things that changed/moved/went away
- [x] Update vty dependency to >= 6.0
- [x] Port the stdin flushing fix patch to vty-unix
vty-windows
- [x] Work with @chhackett to allow time for testing
- [x] Work with @chhackett to get a README added
vty-crossplatform
- [x] Add a README
- [x] Update vty bound to >= 6.0
[x] Coordinate with @glguy on trying out changes with irc-core
[x] Get all-clear on CI build(s)
Hackage releases
- [x] Update brick's latest release metadata to set Vty upper bound to < 6.0
- [x] Tag and release vty 6.0 to Hackage
- [x] Tag and release vty-unix 0.1.0.0 to Hackage
- [x] Work with @chhackett to get vty-windows 0.1.0.0 released to Hackage
- [x] Tag and release vty-crossplatform 0.1.0.0 to Hackage (depends on vty-windows and vty-unix releases)
[x] Prepare public release announcement text
[x] Do public announcement
- [x] https://discourse.haskell.org/
- [x] https://www.reddit.com/r/haskell/
Update brick
- [x] Remove unix dependency
- [x] Merge refactor/vty-crossplatform to master
- [x] Check User Guide for text and code that needs to get updated
- [x] Check README for text that needs to get updated
- [x] Add note about Windows support to README
- [x] Set version to 2.0
- [x] Cut a release to Hackage

Testing your application

Want to test your application before the release? Here’s a high-level summary of what is changing and how to test (full details to follow before the release):

vty will still provide the core rendering functionality and data types and your application will still need to depend on it.
But Vty’s initialization of the terminal (called mkVty) has moved to one of three packages:
- If you only want to support Unix terminals, depend on vty-unix and import mkVty from that package.
- If you only want to support Windows terminals, depend on vty-windows and import mkVty from that package.
- If you want your application to support both platforms and choose the right implementation at build time based on the build environment, depend on vty-crossplatform and import mkVty from that package.

There are some other changes that will likely be necessary, but the amount of changes will depend on how much of the Vty API your application uses. I’m happy to answer questions if you try out the package dependency changes and then run into API issues. For most applications, the upgrade will not involve very many changes, though.

To test your application, git clone the various repositories (and check out the branches mentioned above) and then put the following cabal.project file into your application source directory during testing, updating it to point to the paths to your Git clones:

packages: .
          ../vty
          ../vty-unix
          ../vty-windows
          ../vty-crossplatform

chhackett commented 1 year ago

@jtdaugherty Regarding the first item in your list:

Figure out where the vty demo applications (in demos and examples) should live (probably vty-unix)

I'm interested in your thoughts whether it would be feasible to put the tests, demos, and examples into the crossplatform project instead. My reasoning is that I would like to maintain feature parity across windows and linux. And if I have to try and maintain a separate copy of the tests, that goal is just harder.

I've already got a copy of the demos building and running in the crossplatform project. I'll work on the examples and tests if you are agreeable to this kind of approach.

jtdaugherty commented 1 year ago

'm interested in your thoughts whether it would be feasible to put the tests, demos, and examples into the crossplatform project instead.

I think this could work nicely for the demos and examples since they are end-user programs that depend on a mkVty implementation.

I think the tests are another matter, though; I already split up the tests into those that only test the Vty core internals (those went in vty itself) and those that somehow tested Unix-specific functionality (and those now live in vty-unix). For example, some of the vty-unix tests do some things with terminfo, and I wouldn't assume that those make sense in vty-windows necessarily so I definitely wouldn't want them in vty-crossplatform.

With that said, there could also be tests that make sense in vty-crossplatform if such tests could be written to ensure that the same functionality works as desired on whatever platforms get supported by that package. I don't know what those tests would be at the moment, but I'd be all for adding a test suite to that package in the event that such cases could be devised.

So, I guess what I'm saying is that it seems to me that there could legitimately be tests that belong in each of the four packages, depending on what is being tested.

ShrykeWindgrace commented 1 year ago

Hi!

That's a great write-up, I'll see where and when I can help with these steps this summer.

Regarding announcements, I would say reddit and discourse are two most obvious places to promote the upcoming release. I can also cover Russian-speaking telegram channels. I follow neither IRC nor mailing lists, so I can not say whether they have a significant audience not covered by previous options.

jtdaugherty commented 1 year ago

Thanks, @ShrykeWindgrace!

jtdaugherty commented 1 year ago

@chhackett FYI the Rogue and EventEcho demo programs are now in vty-crossplatform.

ShrykeWindgrace commented 1 year ago

By the way, where do we report issues for current set of packages and integrations? One centralized repository (vty and/or vty-crossplatform) and then redispatch them accordingly? Or dispatch them across the four repositories?

jtdaugherty commented 1 year ago

@ShrykeWindgrace I'd say that if it isn't clear, report on vty. If an issue turns out to be a problem with a different package, we can just migrate it accordingly.

chhackett commented 1 year ago

Just an update for you, @jtdaugherty - I am working on bringing tests into the vty-windows plugin now, analogous to the vty-unix tests. Once I get those working I believe we will be in a position to work on the last few points of your task list here.

I did add a README to vty-windows btw. If you could give it a read and provide some feedback I'd appreciate it. I just started by copying vty README...

hasufell commented 1 year ago

Is there already a way to test this? Does anyone have a cabal.project configuration for this?

chhackett commented 1 year ago

@hasufell Yes, you can certainly test this yourself. And I would appreciate the help. I checked in the test code yesterday.

To run the tests, check out vty and change the branch to refactor/vty-unix-split. Then check out vty-windows (from https://github.com/chhackett/vty-windows.git). The unit tests are in a subproject under the tests directory. My cabal.project config just has the current directory, vty project directory and the vty-windows project directory. So something like ". ../../vty .."

The problem I'm having is with a test that tries to look up and change the console mode to support VT sequences, and is failing with an exception on the GetConsoleMode windows call.

In other contexts, such as running the demo applications (which are provided in the vty-crossplatform project) this all works fine.

hasufell commented 1 year ago

I want to try it in an existing package that depends on vty at the moment. I don't understand the purpose of all those new packages. Can you give a quick migration guide?

jtdaugherty commented 1 year ago

Hi @hasufell, there will definitely be a migration guide before the releases happen, but I have not gotten to that particular task yet. However, here’s a summary:

vty will still provide the core rendering functionality and data types and your application will still need to depend on it.
But Vty’s initialization of the terminal (called mkVty) has moved to one of three packages:
- If you only want to support Unix terminals, depend on vty-unix and import mkVty from that package.
- If you only want to support Windows terminals, depend on vty-windows and import mkVty from that package.
- If you want your application to support both platforms and choose the right implementation at build time based on the build environment, depend on vty-crossplatform and import mkVty from that package.

There are some other changes that will likely be necessary, but the amount of changes will depend on how much of the Vty API your application uses. I’m happy to answer questions if you try out the package dependency changes and then run into API issues. For most applications, the upgrade will not involve very many changes, though.

As far as a cabal.project file, I recommend putting this in your application source directory during testing:

packages: .
          ../vty
          ../vty-unix
          ../vty-windows
          ../vty-crossplatform

And then git clone the various repositories into the parent directory accordingly, being sure to check out the refactor/vty-unix-split branch in the vty clone.

hasufell commented 1 year ago

Is there a patch/branch for brick yet?

chhackett commented 1 year ago

Writing a migration guide is still on our TODO list, but I can give you a quick and dirty version. First, you'll need to add vty-crossplatform as a dependency to your cabal file. Also make sure that your project does not depend on any packages that don't build on Windows, of course (like unix). In your project main module where you construct the 'vty' instance, you'll need to import the new module Graphics.Vty.CrossPlatform.

If your project builds vty with the default configuration, then your project should build and run on Windows as-is.

For an example, see: https://github.com/jtdaugherty/vty-crossplatform/blob/main/programs/ModeDemo.hs

jtdaugherty commented 1 year ago

@chhackett - thanks for your update, that’s great news! The README you added looks good to me.

jtdaugherty commented 1 year ago

@hasufell No, there is not a branch for brick yet, but I’m happy to make one. I’ll get it put up some time today and then I’ll update here.

chhackett commented 1 year ago

@jtdaugherty I do have a locally patched version (of brick) that fixes the one module that breaks on Windows, btw (FileBrowser). Would you like me to submit a PR for that?

jtdaugherty commented 1 year ago

@chhackett yes, please do. I had a note to myself about checking for other Unix dependencies in Brick; you might have fixed the only one.

jtdaugherty commented 1 year ago

@chhackett actually, let me create the Brick branch and then you can submit it against that.

jtdaugherty commented 1 year ago

@chhackett @hasufell here is the brick branch: https://github.com/jtdaugherty/brick/tree/refactor/vty-crossplatform

chhackett commented 1 year ago

PR is up.

jtdaugherty commented 1 year ago

@chhackett Thanks!

ShrykeWindgrace commented 1 year ago

For reference: https://github.com/jtdaugherty/brick/issues/445 =)

hasufell commented 1 year ago

I tried the following cabal.project section:

source-repository-package
  type: git
  location: https://github.com/jtdaugherty/brick.git
  tag: 250c8b471b439950ea4b3eefe45a4cc205755d85

source-repository-package
  type: git
  location: https://github.com/jtdaugherty/vty.git
  tag: 40b2a01d431afdef62a84b8f731cb5846583a088

source-repository-package
  type: git
  location: https://github.com/jtdaugherty/vty-crossplatform.git
  tag: 7beefd86af3386dfbf352563f2211fee94549752

if os(mingw32)
    source-repository-package
        type: git
        location: https://github.com/chhackett/vty-windows.git
        tag: 4cda7fe6f5b92049838e7084795591e09c201131
else
    source-repository-package
        type: git
        location: https://github.com/jtdaugherty/vty-unix.git
        tag: ad31125fe40eb9212fdea8011f34728429d4d213

But it didn't work and vty-unix failed to build:

Configuring library for vty-unix-0.1.0.0..
Preprocessing library for vty-unix-0.1.0.0..
Building library for vty-unix-0.1.0.0..
[ 1 of 18] Compiling Data.Terminfo.Parse ( src/Data/Terminfo/Parse.hs, dist/build/Data/Terminfo/Parse.o, dist/build/Data/Terminfo/Parse.dyn_o )
[ 2 of 18] Compiling Data.Terminfo.Eval ( src/Data/Terminfo/Eval.hs, dist/build/Data/Terminfo/Eval.o, dist/build/Data/Terminfo/Eval.dyn_o )
[ 3 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Classify.Types ( src/Graphics/Vty/Platform/Unix/Input/Classify/Types.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Classify/Types.o, dist/build/Graphics/Vty/Platform/Unix/Input/Classify/Types.dyn_o )
[ 4 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Classify.Parse ( src/Graphics/Vty/Platform/Unix/Input/Classify/Parse.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Classify/Parse.o, dist/build/Graphics/Vty/Platform/Unix/Input/Classify/Parse.dyn_o )
[ 5 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Focus ( src/Graphics/Vty/Platform/Unix/Input/Focus.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Focus.o, dist/build/Graphics/Vty/Platform/Unix/Input/Focus.dyn_o )
[ 6 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Mouse ( src/Graphics/Vty/Platform/Unix/Input/Mouse.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Mouse.o, dist/build/Graphics/Vty/Platform/Unix/Input/Mouse.dyn_o )
[ 7 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Paste ( src/Graphics/Vty/Platform/Unix/Input/Paste.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Paste.o, dist/build/Graphics/Vty/Platform/Unix/Input/Paste.dyn_o )
[ 8 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Classify ( src/Graphics/Vty/Platform/Unix/Input/Classify.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Classify.o, dist/build/Graphics/Vty/Platform/Unix/Input/Classify.dyn_o )
[ 9 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Terminfo.ANSIVT ( src/Graphics/Vty/Platform/Unix/Input/Terminfo/ANSIVT.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Terminfo/ANSIVT.o, dist/build/Graphics/Vty/Platform/Unix/Input/Terminfo/ANSIVT.dyn_o )
[10 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Terminfo ( src/Graphics/Vty/Platform/Unix/Input/Terminfo.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Terminfo.o, dist/build/Graphics/Vty/Platform/Unix/Input/Terminfo.dyn_o )
[11 of 18] Compiling Graphics.Vty.Platform.Unix.Output.Color ( src/Graphics/Vty/Platform/Unix/Output/Color.hs, dist/build/Graphics/Vty/Platform/Unix/Output/Color.o, dist/build/Graphics/Vty/Platform/Unix/Output/Color.dyn_o )
[12 of 18] Compiling Graphics.Vty.Platform.Unix.Output.TerminfoBased ( src/Graphics/Vty/Platform/Unix/Output/TerminfoBased.hs, dist/build/Graphics/Vty/Platform/Unix/Output/TerminfoBased.o, dist/build/Graphics/Vty/Platform/Unix/Output/TerminfoBased.dyn_o )

src/Graphics/Vty/Platform/Unix/Output/TerminfoBased.hs:209:39: warning: [-Wdeprecations]
    In the use of ‘tiGetStr’
    (imported from System.Console.Terminfo, but defined in System.Console.Terminfo.Base):
    Deprecated: "use tiGetOutput instead."
    |
209 |     = case Terminfo.getCapability ti (Terminfo.tiGetStr capName) of
    |                                       ^^^^^^^^^^^^^^^^^

src/Graphics/Vty/Platform/Unix/Output/TerminfoBased.hs:215:39: warning: [-Wdeprecations]
    In the use of ‘tiGetStr’
    (imported from System.Console.Terminfo, but defined in System.Console.Terminfo.Base):
    Deprecated: "use tiGetOutput instead."
    |
215 |     = case Terminfo.getCapability ti (Terminfo.tiGetStr capName) of
    |                                       ^^^^^^^^^^^^^^^^^
[13 of 18] Compiling Graphics.Vty.Platform.Unix.Output.XTermColor ( src/Graphics/Vty/Platform/Unix/Output/XTermColor.hs, dist/build/Graphics/Vty/Platform/Unix/Output/XTermColor.o, dist/build/Graphics/Vty/Platform/Unix/Output/XTermColor.dyn_o )
[14 of 18] Compiling Graphics.Vty.Platform.Unix.Settings ( src/Graphics/Vty/Platform/Unix/Settings.hs, dist/build/Graphics/Vty/Platform/Unix/Settings.o, dist/build/Graphics/Vty/Platform/Unix/Settings.dyn_o )
[15 of 18] Compiling Graphics.Vty.Platform.Unix.Output ( src/Graphics/Vty/Platform/Unix/Output.hs, dist/build/Graphics/Vty/Platform/Unix/Output.o, dist/build/Graphics/Vty/Platform/Unix/Output.dyn_o )
[16 of 18] Compiling Graphics.Vty.Platform.Unix.Input.Loop ( src/Graphics/Vty/Platform/Unix/Input/Loop.hs, dist/build/Graphics/Vty/Platform/Unix/Input/Loop.o, dist/build/Graphics/Vty/Platform/Unix/Input/Loop.dyn_o )
[17 of 18] Compiling Graphics.Vty.Platform.Unix.Input ( src/Graphics/Vty/Platform/Unix/Input.hs, dist/build/Graphics/Vty/Platform/Unix/Input.o, dist/build/Graphics/Vty/Platform/Unix/Input.dyn_o )
[18 of 18] Compiling Graphics.Vty.Platform.Unix ( src/Graphics/Vty/Platform/Unix.hs, dist/build/Graphics/Vty/Platform/Unix.o, dist/build/Graphics/Vty/Platform/Unix.dyn_o )
Warning: the following files would be used as linker inputs, but linking is not being done: cbits/gwinsz.h
ghc: no input files
Usage: For basic information, try the `--help' option.
Error: cabal: Failed to build vty-unix-0.1.0.0

This is with cabal-install-3.10.1.0.

hasufell commented 1 year ago

It seems to work with cabal-install-3.6.2.0 though, so it looks like a cabal regression.

jtdaugherty commented 1 year ago

@hasufell thanks for your testing. Were there any other issues once Cabal build vty-unix successfully?

hasufell commented 1 year ago

@hasufell thanks for your testing. Were there any other issues once Cabal build vty-unix successfully?

Doesn't seem so. But I have not built on windows yet.

jtdaugherty commented 11 months ago

@chhackett on your comment above about the vty-windows README, I took a look. I think the only thing I'd recommend is to mention that vty-crossplatform might be a preferable dependency if someone wants to support multiple platforms. vty-windows itself would only be recommended for developers who only want to support Windows. So maybe add a note about that with a link to the repo. (By the time you release your package, I'll be about to release vty-crossplatform so it'll be on Hackage by that time.)

jtdaugherty commented 10 months ago

My release announcement text draft:

I’m happy to announce the release of vty version 6.0. The highlight
of this release is that vty now works on Windows, thanks to hard work
by Chris Hackett and Timofey Zakrevskiy! For years, many people in the
Haskell community have requested Windows support in vty. Thanks to
Chris and Timofey for the time and energy they put into this, and their
patience as we worked out an approach! Thanks also to Eric Mertens
and Kevin Quick for early feedback on the release.

In this release, the core "vty" package was reorganized so that support
for specific platforms now lives in platform-specific packages that need
to be used alongside the core "vty" package. This release includes the
release of three aditional new packages:

* vty-unix (Unix terminal support previously provided by "vty")
* vty-windows (the Windows backend)
* vty-crossplatform (for when you want to support both)

The core package still provides all of the important library
functionality and is still a necessary dependency. Applications will
need to depend on an additional package for the platform that they need
to support. When in doubt, use vty-crossplatform!

The release changelog update has a detailed listing of changes as well
as instructions for how to update your application to work with Vty 6:

https://github.com/jtdaugherty/vty/blob/master/CHANGELOG.md#60

To get help, please open an issue on the Vty repository.

While I'm at it, Brick has been updated to use vty-crossplatform in version 2.0!

Thanks!

hasufell commented 10 months ago

Cool, ghcup might be the first to use it for next release.

jtdaugherty commented 10 months ago

@hasufell I'm glad to hear it! I'm currently working with @chhackett to choose a release date. I'll update here when we have a date in mind.

jtdaugherty commented 10 months ago

@chhackett I'm wondering if you could stress-test ghcup on a Windows system. Do you have time to give that a try?

ShrykeWindgrace commented 10 months ago

I tried building ghcup (without tui) on windows with stack and failed. Maybe cabal can manage that.

ShrykeWindgrace commented 10 months ago

My release announcement text draft:

Hurrah!

hasufell commented 10 months ago

I tried building ghcup (without tui) on windows with stack and failed. Maybe cabal can manage that.

Use cabal --project-file=cabal.project.release build -w ghc-9.2.8 (with cabal-3.10.1.0)

ShrykeWindgrace commented 10 months ago

I tried building ghcup (without tui) on windows with stack and failed. Maybe cabal can manage that.

Use cabal --project-file=cabal.project.release build -w ghc-9.2.8 (with cabal-3.10.1.0)

I'll test that, thanks=)

jtdaugherty commented 10 months ago

FYI: @chhackett and I are planning on October 30 for the releases.

jtdaugherty commented 10 months ago

As of around 3 PM Pacific time today, vty 6.0, vty-crossplatform, and vty-unix have been published to Hackage. Once vty-windows is uploaded, I'll put up the announcements.

chhackett commented 10 months ago

So I ran into a permission problem. Apparently I need to be added to the 'upload group' of hackage. I did not realize there was more to the process beyond getting an account.

So I sent out the email to request that, but I guess I don't know how long it will take to get that issue resolved.

I guess I'm a little surprised by this. I read the page about uploading (https://hackage.haskell.org/upload) and it just said I needed a username/password, which I got.

jtdaugherty commented 10 months ago

@chhackett ah yeah, it's been so long since I had to do that that I forgot that it was a thing. No problem!

chhackett commented 10 months ago

I'll let you know here as soon as I get permission, and get vty-windows uploaded.

chhackett commented 10 months ago

vty-windows is uploaded. Thank you @jtdaugherty for orchestrating all the pieces of this project, and @ShrykeWindgrace for all your help with the prototyping, investigation and problem solving skills.

jtdaugherty commented 10 months ago

Announcements are now up!

Thanks, everyone!

And with that, I will now close this ticket!

simonmichael commented 10 months ago

Huge thanks to all who made this happen! While I'm not using Windows myself, I very much appreciate that this allows my vty- and brick-based software to also run on Windows with little effort on my part. 🙏🏻

jtdaugherty / vty

Preparing for vty multi-platform release #260

Testing your application