search

plotly / Kaleido

Fast static image export for web-based visualization libraries with zero dependencies
Other
348 stars 33 forks source link

Fix Windows Build #179

Closed ayjayt closed 5 days ago

ayjayt commented 2 months ago

Win build is fixed for Chromium Stable 88- the last one plotly released.

Highlighted wins:

What's Next

"Urgent" Technical Issues

- Migrate Tooling

Use make, bash, rsync for everything. Move away from scripts except for CI. Unify interface between platforms, factor out common code, and make interface should be:

make set_version # choose chromium version, choose platform, everything else is auto
make fetch_tools # git clone depot_tools
make init_tools #  look at note in fetch_chromium.ps1, info changes based on version/platform?
make sync # runs gclient
make gen_meta # CREDITS, license.py, version, README, etc
make patch # runs patches based on versions
make config_build # appends kaleido instructions to `BUILD.gn`, runs `gn gen out`
make compile # runs ninja
make extract_build # changes based on version! see next point as well. also does mathjax, etc.
make npm # does npm stuff
make javascript # copies npm stuff
make wheel # duh
# fun fact, the zip part is unecessary because wheels actually are `.zip` files of python packages renamed to `.whl`...

Reorganizing the repository would come along with this. It's really only a days labor, especially with this outline.

The problem is 'version.py' constantly changes trivial code and forces recompile of 5k objects. Mal!

For the CI, it's a simple as skipping make set_version which really just sets environmental variables and then running make all. Boom!

- Determine run-time dependencies

We have this annoying issue where even ldd won't show us which dll/dynlib/so are needed in the wheel because they appear to be linked dynamically. So for every version, given that outputs and chromium needs change, we need to determine which outputs to our compile actually need to come with us (LOTS do not). The solution for this is as follows:

  1. Python is a shell for kaleido, and kaleido and chromium should optionally report all errors back to the python.
  2. Test kaleido against ALL mocks.
  3. Tests kick back if chromium throws an error to python.

- Remove dependency on base::... just to reduce maintenance

- Update Chromium API... no other option, they 86'ed it.

Longer Discussion

What I did:

1) We don't download the entire chromium repo anymore, just the revevant tag. 30GB in savings.

2) Errors were fixed in power shell scripts (using what looked like bash)

3) Power shell scripts now fail loudly instead of pretending to succeed.

4) Exhaustive build documentation, but to the point and separate from user documentation (and also divided by platform).

5) Got Chromium 88 to build for windows, had to write two patches.

6) Got Chromium 108 to build, but has runtime errors.

Unfortunately, the API used in kaleido was officially removed after version 108, and had been degrading up until that point. Migrating to the current API is a bit of a job.

Also, we're relying on chromiums base:: namespace, which they don't recommend. It is unstable, but I did update it to work to get version 108 to build. Version 88 uses old version of base::. A long-lasting solution would actually minimize exposure to chromium APIs, ironically.

All this so my users can make videos like this:

https://github.com/plotly/Kaleido/assets/30324885/dc13af46-5e78-4c4d-a03c-94d6cc40e554

Pretty, aint it?

ayjayt commented 2 months ago

Interestingly, kaleido==0.2.1 works on some windows machines over here, but not mine. So far my fix works on all ours.

Here is mine: pip install --index-url https://test.pypi.org/simple/ kaleidowinfix2

Normally don't recommend installing exe's from randos.

@jonmmease, just tagging you for visibility. thanks for getting all this started.

@archmoj could you tag anyone who might be interested or benefit from this at plotly? i would like to coordinate a re-release and i have no experience with circle ci.

@Coding-with-Adam tagging you because there is a bunch of community questions about this, tons on github and a couple in the forums. i know you're waiting for c2m but this was blocking 3 people over here so I had to do it.

ndrezn commented 2 months ago

Hi @ayjayt! Thanks for this contribution! We're taking a look internally and looking to set time to review this PR.

ayjayt commented 2 months ago

Hi, thanks.

FYI, it's 99% build stuff.

I am planning on another PR probably within two weeks, that will be a bigger step forward.

Happy to set up meeting/presentation at that point.

El jue, 2 de may. de 2024 10:28 a. m., Nathan Drezner < @.***> escribió:

Hi @ayjayt https://github.com/ayjayt! Thanks for this contribution! We're taking a look internally and looking to set time to review this PR.

— Reply to this email directly, view it on GitHub https://github.com/plotly/Kaleido/pull/179#issuecomment-2090809474, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHHLRFNERYXR37KVXUB4ZQLZAJLRZAVCNFSM6AAAAABGSDGA72VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJQHAYDSNBXGQ . You are receiving this because you were mentioned.Message ID: @.***>

gvwilson commented 1 month ago

thanks very much for this @ayjayt - any chance you could spare a few minutes next week to chat online? cheers - @gvwilson

ayjayt commented 1 month ago

Sure @gvwilson, things are pretty wild over here on my end right now but I can make time for a chat. ajpikul@gmail.com or however you'd like.