Include JS source code in source distribution (sdist)

[thewchan]

I was able to the v2.0.2 up to conda forge, but the most recent version seems to build JS code as well, but the JS code isn't shipped with the sdist and therefore we aren't able to build the most recent tldraw verison on Conda Forge until the JS source code is included in the sdist.

[thewchan]

Hi pinging for this again, we have other packages depending on tldraw on Conda @kolibril13

[kolibril13]

hi @thewchan, I won't have much time to invest in this, but can you give me more insights how to ship js to sdist?

Currently, I'm using hatch build with this configuration to build the dist folder which is then uploaded to pypi.

When I unpack the whl file, you will find the js files in src/tldraw/static

[thewchan]

Hi @kolibril13 thanks for taking the time; it seems that the JS files needed to build the wheel is not only the src/static ones, but rather also the ones in the js path outside of src. Here's the error I got when the CI is trying to build the package with the SDIST on pypi:

 Running command Preparing metadata (pyproject.toml)
  INFO:hatch_jupyter_builder.utils:Running jupyter-builder
  INFO:hatch_jupyter_builder.utils:Building with hatch_jupyter_builder.npm_builder
  INFO:hatch_jupyter_builder.utils:With kwargs: {'npm': 'npm', 'build_cmd': 'build', 'path': 'js'}
  INFO:hatch_jupyter_builder.utils:Installing build dependencies with npm.  This may take a while...
...
FileNotFoundError: [Errno 2] No such file or directory: '/home/conda/feedstock_root/build_artifacts/tldraw_1720629166140/work/js'

I think it's looking for the JS files in this location: https://github.com/kolibril13/jupyter-tldraw/tree/main/js Which is not included in the SDIST.

I did a quick search and it seems that it's not too difficult to include it via hatch/pyproject.toml: https://hatch.pypa.io/latest/config/build/#patterns

And I think this hatch gh issue addresses it: https://github.com/pypa/hatch/issues/492. Thanks again for checking this out!

[kolibril13]

ok, I just made this change and released version 2.2.5 https://pypi.org/project/tldraw/#history that include all the jsx files

[thewchan]

@kolibril13 thanks so much I'm almost able to get this to work; it seems that it's missing package.json file in the root directory:

  Running command Preparing metadata (pyproject.toml)
  INFO:hatch_jupyter_builder.utils:Running jupyter-builder
  INFO:hatch_jupyter_builder.utils:Building with hatch_jupyter_builder.npm_builder
  INFO:hatch_jupyter_builder.utils:With kwargs: {'npm': 'npm', 'build_cmd': 'build', 'path': 'js'}
  INFO:hatch_jupyter_builder.utils:Installing build dependencies with npm.  This may take a while...
  INFO:hatch_jupyter_builder.utils:> /home/conda/feedstock_root/build_artifacts/tldraw_1720717706280/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin/npm install
  npm error code ENOENT
  npm error syscall open
  npm error path /home/conda/feedstock_root/build_artifacts/tldraw_1720717706280/work/js/package.json
  npm error errno -2
  npm error enoent Could not read package.json: Error: ENOENT: no such file or directory, open '/home/conda/feedstock_root/build_artifacts/tldraw_1720717706280/work/js/package.json'
  npm error enoent This is related to npm not being able to find a file.
  npm error enoent
  npm error A complete log of this run can be found in: /home/conda/.npm/_logs/2024-07-11T17_10_43_257Z-debug-0.log

Basically, the way conda build works is it downloads the sdist file from PyPI, so only the files included in that sdist archive file is available to build the wheel file in Conda's CI pipeline. Hopefully this is the last hurdle!

[kolibril13]

https://pypi.org/project/tldraw/2.2.6/ Just made this release:

[thewchan]

@kolibril13 thanks so much for you help and I'm so sorry this is turning into such a problem for me. It seems the JS code is able to begin building, but it seems like I'm running into some path issues, it's not able to find the imported nodejs packages it seems? Do you have any ideas what might be causing this?

  INFO:hatch_jupyter_builder.utils:Building with hatch_jupyter_builder.npm_builder
  INFO:hatch_jupyter_builder.utils:With kwargs: {'npm': 'npm', 'build_cmd': 'build', 'path': 'js'}
  INFO:hatch_jupyter_builder.utils:Installing build dependencies with npm.  This may take a while...
  > node node_config.mjs

  ✘ [ERROR] Could not resolve "react"

      js/debug.jsx:1:23:
        1 │ import * as React from "react";
          ╵                        ~~~~~~~

    The module "./index.js" was not found on the file system:

      node_modules/react/package.json:25:17:
        25 │       "default": "./index.js"
           ╵                  ~~~~~~~~~~~~

    You can mark the path "react" as external to exclude it from the bundle, which will remove this error and leave the unresolved path in the bundle.

  ✘ [ERROR] Could not resolve "@anywidget/react"

      js/debug.jsx:3:44:
        3 │ import { createRender, useModelState } from "@anywidget/react";
          ╵                                             ~~~~~~~~~~~~~~~~~~

    The module "./index.js" was not found on the file system:

      node_modules/@anywidget/react/package.json:11:16:
        11 │       "import": "./index.js"
           ╵                 ~~~~~~~~~~~~

    You can mark the path "@anywidget/react" as external to exclude it from the bundle, which will remove this error and leave the unresolved path in the bundle.

  ✘ [ERROR] Could not resolve "@tldraw/tldraw/tldraw.css"

      js/debug.jsx:4:7:
        4 │ import "@tldraw/tldraw/tldraw.css";
          ╵        ~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

If you have any idea I might be able to fix this on my side in the CI.

[kolibril13]

hmm, no idea what might cause this problem I'm just importing react like this import * as React from "react"; and npm takes care of the rest. The expert in anywidget and import paths is @manzt , maybe he has insights on how to bring anywidget-react projects to anaconda.

[thewchan]

Thanks @manzt and @kolibril13, some older versions of tldraw is already available on Conda Forge (community maintained repo of packages installable via conda), I'm trying to figure out why the newer versions won't build correctly in our CI.

Here's where I'm trying to do so: https://github.com/conda-forge/tldraw-feedstock/pull/12/

I think you all would still be able to see the CI output, any help greatly appreciated here!

[thewchan]

@kolibril13 We've figuring this out! The JS packages path got a little confused because npm installed packages in the CI, but also the sdist bundled a node_modules folder, if we remove that folder in the build CI the package is able to build successfully. We do suggest in future releases to unbundle the node_modules folder from the sdist (possibly using the ! operator in the artifacts configuration in pyproject.toml, see: https://hatch.pypa.io/latest/config/build/#artifacts), but this is not currently blocking the build of tldraw in Conda.

Thanks so much again for your support @kolibril13!