search

bblanchon / pdfium-binaries

📰 Binary distribution of PDFium
789 stars 166 forks source link

Experimental conda packaging (work in progress) #120

Closed mara004 closed 8 months ago

bblanchon commented 8 months ago

Thanks! That's a good starting point. However, I think I'll do the conda build in a dedicated job instead of adding it to the build script. I'll give it a try and get back to you.

mara004 commented 8 months ago

However, I think I'll do the conda build in a dedicated job instead of adding it to the build script.

Sure, do as you like. This is only a draft/demo after all.

I guess you intend to do all the conda packaging on the root runner in build-all.yml (as for nuget), right? This will take some different logic than if integrated into the build steps and also different source path handling in the recipe, but I think you're right it will be cleaner this way.

bblanchon commented 8 months ago

I tried to make sense of this, but I need clarification on all the folder manipulation. How can I build the one package that includes all the binaries, given that all the required files are on disk? It should just be a call to conda build, nothing else, right?

mara004 commented 8 months ago

I tried to make sense of this, but I need clarification on all the folder manipulation. How can I build the one package that includes all the binaries, given that all the required files are on disk? It should just be a call to conda build, nothing else, right?

I'm not sure what you mean?

With conda we build one package for each platform, not a single big package for all platforms. For packages that match the host platform it will be just a call to conda build; for non-host packages we have to create an intermediary fake host package, then convert it to the target using conda convert.

mara004 commented 8 months ago

Also see my edited post above regarding what I believe you intend to do. I think a key point is the source path in the recipe. If you want to build all packages on the root runner, it will have to be made dynamic through an env var.

bblanchon commented 8 months ago

I made some progress, as you can see on this build: https://github.com/bblanchon/pdfium-binaries/actions/runs/5972658236/job/16203843717

I'm facing issues with the Windows build, and I wonder if creating a Windows package from a Linux host is really supported.

If I copy pdfium.dll in bin/ here is what I get:

+ conda convert -f -p win-32 out/linux-64/pdfium-binaries-118.0.5961-0.tar.bz2 -o staging/
Converting pdfium-binaries-118.0.5961-0.tar.bz2 from linux-64 to win-32

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.11/site-packages/conda/exception_handler.py", line 17, in __call__
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda/cli/main.py", line 64, in main_subshell
        exit_code = do_call(args, parser)
                    ^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 143, in do_call
        result = plugin_subcommand.action(getattr(args, "_args", args))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda_build/plugin.py", line 16, in convert
        execute(*args, **kwargs)
      File "/opt/conda/lib/python3.11/site-packages/conda_build/cli/main_convert.py", line 129, in execute
        api.convert(f, **args.__dict__)
      File "/opt/conda/lib/python3.11/site-packages/conda_build/api.py", line 404, in convert
        return conda_convert(
               ^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 874, in conda_convert
        convert_from_unix_to_windows(
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 732, in convert_from_unix_to_windows
        update_paths_file(temp_dir, target_platform="win")
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 415, in update_paths_file
        path["sha256"] = update_executable_sha(temp_dir, path["_path"])
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 349, in update_executable_sha
        with open(os.path.join(package_directory, executable_path), "rb") as script_file:
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpv93itaar/Scripts/pdfium-script.py'

If I copy pdfium.dll in lib/ here is what I get:

+ conda convert -f -p win-32 out/linux-64/pdfium-binaries-118.0.5961-0.tar.bz2 -o staging/
Converting pdfium-binaries-118.0.5961-0.tar.bz2 from linux-64 to win-32

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/opt/conda/lib/python3.11/site-packages/conda/exception_handler.py", line 17, in __call__
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda/cli/main.py", line 64, in main_subshell
        exit_code = do_call(args, parser)
                    ^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 143, in do_call
        result = plugin_subcommand.action(getattr(args, "_args", args))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda_build/plugin.py", line 16, in convert
        execute(*args, **kwargs)
      File "/opt/conda/lib/python3.11/site-packages/conda_build/cli/main_convert.py", line 129, in execute
        api.convert(f, **args.__dict__)
      File "/opt/conda/lib/python3.11/site-packages/conda_build/api.py", line 404, in convert
        return conda_convert(
               ^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 874, in conda_convert
        convert_from_unix_to_windows(
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 732, in convert_from_unix_to_windows
        update_paths_file(temp_dir, target_platform="win")
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 409, in update_paths_file
        path["_path"] = update_lib_path(path["_path"], "win")
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 243, in update_lib_path
        python_version = retrieve_python_version(path)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/opt/conda/lib/python3.11/site-packages/conda_build/convert.py", line 108, in retrieve_python_version
        with open(path_file) as index_file:
             ^^^^^^^^^^^^^^^
    FileNotFoundError: [Errno 2] No such file or directory: 'lib/pdfium.dll/info/index.json'

Anyway, I spent way too much time on this, and I need to get back to work for the moment.

bblanchon commented 8 months ago

I am tired of wasting time on these awful tools (conda build and conda convert). What a mess to create a tarball!

I would probably be better off building the tarball from a script since the specification is available. What do you think?

mara004 commented 8 months ago

Anyway, I spent way too much time on this, and I need to get back to work for the moment.

I am tired of wasting time on these awful tools (conda build and conda convert). What a mess to create a tarball!

I'm sorry for the inconveniences; I too have been feeling this way. It is a mess indeed, these tools are slow and not well designed. conda packaging just seems to be very centered towards the native hosts model of anaconda's CI... Nevertheless, I would like to finally get this through because we've put in a lot of effort already.

I would probably be better off building the tarball from a script since the specification is available. What do you think?

I've once thought about this too, but frustration aside, I don't think it's reasonable. Probably this would be way more work and sufficiently difficult to get right (and maintain).

I'm facing issues with the Windows build, and I wonder if creating a Windows package from a Linux host is really supported.

If it doesn't work it probably isn't supported, unfortunately. How about just building the windows packages natively on windows and use the bld.bat as shown in this PR? Maybe integrating this into the build steps as I initially thought was better after all?

(FWIW converting from linux to windows did work with python though, see pypdfium2's conda bundling experiment)

BTW I like the -o staging/, this avoids some of my strange directory logic.

mara004 commented 8 months ago

Ideally the recipe should just provide an option to set the platform directly and include files in an OS independent manner, but I couldn't find how, so probably this is not supported.

bblanchon commented 8 months ago

How about just building the windows packages natively on windows

I prefer not to, because:

  1. I don't want conda to be required to build other packages
  2. I don't want the build to fail if there is a problem with conda
  3. I want a dedicated job to upload to anaconda, so it makes sense to build the packages there
  4. I want to keep each task isolated (imagine if we did the same with all packages formats)
  5. I'm using the same pattern for NuGet and it's perfect

BTW, I also tied the undocumented command conda package, which looked promising, but I could not understand how it's supposed to work.

mara004 commented 8 months ago

OK, I certainly agree with that rationale, it should not be part of the main build. However, we can still divide the isolated receiver job into native windows/linux/mac hosts, no? Not very nice, I agree, but should be technically possible?

bblanchon commented 8 months ago

Not very nice, I agree, but should be technically possible?

Sure, but is this better than writing our own packaging script, I'm not sure.

BTW, after we have the packages, how are we supposed to upload them? Please don't tell me a Linux machine cannot upload Windows packages 🤪

mara004 commented 8 months ago

BTW, after we have the packages, how are we supposed to upload them? Please don't tell me a Linux machine cannot upload Windows packages 🤪

I don't think so, or at least I hope 😅

I guess we need to gather the packages on one runner and then probably something like ANACONDA_API_TOKEN=... anaconda upload $PACKAGES_DIR/*.tar.bz2. Or alternatively I think there is also conda repo upload ... (assuming conda-repo-cli is installed). Probably setup-miniconda makes both available out of the box.

Uploading the packages should really be the lesser problem.

(Sorry again for the trouble. Disclaimer: I neither use nor like conda. Just want to do it in favor of the users who are requesting it.)

mara004 commented 8 months ago

Sure, but is this better than writing our own packaging script, I'm not sure.

Perhaps you're right, the conda2 branch looks promising!