Unified Installer discussion

[bmaltais]

This issue is to work out bugs/enhancements related to the newly introduced Universal Installer. This installer will support installation ob Linux, MacOS and Windows.

Please give it a try as we need multiple testers to validate that the script perform as well or better than the previous solutions on all platforms. THe user experience is also significantly changed.

Give us feedback on what you think, what worke well and what could be improved.

Huge thanks to @jstayco for his hard work on this. He is a great example of how community comes together to improve projects and move them forward for the benefits of others!

You can join the discussion about how to test and report results here also: https://github.com/bmaltais/kohya_ss/discussions/715

[bmaltais]

@jstayco I tried training with AdamW8bit on windows and got a bitsandbytes error. Is it possible the bitsandbytes files are not getting copied by the launcher?

Hummm... don't think that is the issue... weird... tried manually copying them and still fail with:

================================================================================
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
D:\kohya_ss\venv\lib\site-packages\bitsandbytes\cuda_setup\paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')}
  warn(
WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
CUDA SETUP: Loading binary D:\kohya_ss\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so...
Traceback (most recent call last):
  File "D:\kohya_ss\train_network.py", line 773, in <module>
    train(args)
  File "D:\kohya_ss\train_network.py", line 211, in train
    optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
  File "D:\kohya_ss\library\train_util.py", line 2528, in get_optimizer
    import bitsandbytes as bnb
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\autograd\_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 41, in <module>
    lib = CUDALibrary_Singleton.get_instance().lib
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 37, in get_instance
    cls._instance.initialize()
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 31, in initialize
    self.lib = ct.cdll.LoadLibrary(binary_path)
  File "C:\Users\berna\AppData\Local\Programs\Python\Python310\lib\ctypes\__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "C:\Users\berna\AppData\Local\Programs\Python\Python310\lib\ctypes\__init__.py", line 364, in __init__
    if '/' in name or '\\' in name:
TypeError: argument of type 'WindowsPath' is not iterable

[bmaltais]

I think I found the issue... the wrong version of bitsandbytes is installes... will push an update for that,,, No, that is not that... the version is proper... 0.35.0 so something else

[jstayco]

For your reference, if you need to change file paths for the bitsandbytes library copying it's controlled right here: https://github.com/bmaltais/kohya_ss/blob/dev/launcher.py#L1257

Otherwise? Bump versions like normal in requirements.txt and should be all good.

[bmaltais]

OK... I think launcher.py is not copying the bitsandbytes files as it should. I "properly" manually copied the files this time and now the error is different...

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
CUDA SETUP: Loading binary D:\kohya_ss\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll...
Traceback (most recent call last):
  File "D:\kohya_ss\train_network.py", line 773, in <module>
    train(args)
  File "D:\kohya_ss\train_network.py", line 211, in train
    optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
  File "D:\kohya_ss\library\train_util.py", line 2528, in get_optimizer
    import bitsandbytes as bnb
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\autograd\_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 43, in <module>
    lib = CUDALibrary_Singleton.get_instance().lib
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 39, in get_instance
    cls._instance.initialize()
  File "D:\kohya_ss\venv\lib\site-packages\bitsandbytes\cextension.py", line 33, in initialize
    self.lib = ct.cdll.LoadLibrary(str(binary_path))            # $$$
  File "C:\Users\berna\AppData\Local\Programs\Python\Python310\lib\ctypes\__init__.py", line 452, in LoadLibrary
    return self._dlltype(name)
  File "C:\Users\berna\AppData\Local\Programs\Python\Python310\lib\ctypes\__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 193] %1 is not a valid Win32 application

No idea why it is failing now ;-(

I moved back to the pre-unified_installer_merge branch and re-ran the old setup and training with AdamW8bit worked as expected... so most certainly a difference on how packages are installed is causing this... but what...

[jstayco]

Could you get me a list of files that are supposed to be installed if its bitsandbytes stuff missing? I'm looking at the source and destination in the linked file above and it passes the common sense test.

Maybe a list of files in the working version of this folder: D:\kohya_ss\venv\lib\site-packages\bitsandbytes\ Versus the nonworking one would tell us what's not copying right if that's the issue?

[bmaltais]

Are the bitsandbytes files copied before the requirements files are installed or after? THe function name lead me to beleive they are installed before... when they need to be installed after?

[jstayco]

To be honest I was just as confused, but that order of operations came from one of the early docker implementations that was presented to us. It should definitely be after, right? If you want an easy quick test just switch the function order in main().

[bmaltais]

I tested moving it after and it fixed that issue... but now there is a new one:

use 8-bit AdamW optimizer | {}
Traceback (most recent call last):
  File "D:\kohya_ss\train_network.py", line 773, in <module>
    train(args)
  File "D:\kohya_ss\train_network.py", line 211, in train
    optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
  File "D:\kohya_ss\library\train_util.py", line 2532, in get_optimizer
    optimizer_class = bnb.optim.AdamW8bit
AttributeError: module 'bitsandbytes' has no attribute 'optim'
Traceback (most recent call last):
  File "C:\Users\berna\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\berna\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
  File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main
    args.func(args)
  File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 923, in launch_command
    simple_launcher(args)
  File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 579, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)

[jstayco]

I'll patch the order of ops on that one. My best guess there is that if an attr wasn't found, maybe a .dll is missing outside of the ones we copy?

[bmaltais]

No, all the 4 files are there... the files were copied before bitsandbytes was installed and were overwritten when pip installed the module...

[bmaltais]

I really don't understand why things work fine then I go back to the pre- branch but fail... I did a diff between the pip freeze output and there is hardly any differences.

[bmaltais]

I don't know. THe branch with the unified installer just won't train in 8bit mode. No idea why that is...

[jstayco]

Do you think you could get me a pip list + file tree of the venv folder? I'll try to run the install on the Windows machine I have and see if I can spot the difference. The command to run in cmd for this is tree /a /f > file_tree.doc. You run that command from the venv's site-packages folder to get the relevant files.

[jstayco]

Another question is do you see the same behavior on Torch 1 and Torch 2? Or just one in particular?

[wzgrx]

On wind11 toctorch==2.0.0+cu118 torchvision==0.15.1+cu118 xformers》=0.0.17rc482 TypedStorage is deprecated error will occur

[bmaltais]

@jstayco THe same issue exist for both v1 and v2. I have a feeling it is related to the modifications dome to the kohya_ss.py script but I am not 100% sure. I will try to replace the code with the original one (and all the library code also) and see if it goes away.

Here is the error for Torch v1:

Deprecated: use prepare_optimizer_params(text_encoder_lr, unet_lr, learning_rate) instead of prepare_optimizer_params(text_encoder_lr, unet_lr)
use 8-bit AdamW optimizer | {}
Traceback (most recent call last):
  File "D:\kohya_ss\train_network.py", line 773, in <module>
    train(args)
  File "D:\kohya_ss\train_network.py", line 211, in train
    optimizer_name, optimizer_args, optimizer = train_util.get_optimizer(args, trainable_params)
  File "D:\kohya_ss\library\train_util.py", line 2532, in get_optimizer
    optimizer_class = bnb.optim.AdamW8bit
AttributeError: module 'bitsandbytes' has no attribute 'optim'
Traceback (most recent call last):
  File "C:\Users\berna\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\berna\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\kohya_ss\venv\Scripts\accelerate.exe\__main__.py", line 7, in <module>
  File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main
    args.func(args)
  File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 923, in launch_command
    simple_launcher(args)
  File "D:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 579, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)

[bmaltais]

OK, I ran an experiment. I applied only a few files from the unified installer on top of the working pre-Unified_installer branch to see if things would work again and they did. Here are the files I copied:

• kohya_gui.py
• launcher.py
• setup.bat
• setup.ps1
• setup.sh
• install_config.yml
• requirements_macos.txt

The installer did not properly copy the 4 bitsandbytes files so I had to re-copy them.

Then running the GUI I was able to train with no issue... So the problem must be related to other files that were not supposed to be updated... I feel like replacing all the files in the current pre-unified_installer branch with this minimalist working set of files so we get back to a working state?

[bmaltais]

If you want to test the minimal set of files to compare you can checkout branch test_unified_installer

[wzgrx]

If you want to test the minimal set of files to compare you can checkout branch test_unified_installer

I tried, there were many errors reported

y installing, you accept licenses for the packages. python v3.10.0 already installed. Use --force to reinstall, specify a version to install, or try upgrade.

Chocolatey installed 0/1 packages. See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

Warnings:

• python - python v3.10.0 already installed. Use --force to reinstall, specify a version to install, or try upgrade. Install-Python310 : Failed to install. Please ensure Python 3.10 is installed and available in $PATH. 所在位置 G:\Soft\test\kohya_ss\setup.ps1:2181 字符: 17
  • Install-Python310 -Interactive:$Params.interactive
  • CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
  • FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Install-Python310

PowerShell script failed or was blocked. Falling back to Python script.

WARNING: This MSI requires uninstall prior to installing a different version. Please ask the package maintainer(s) to add a check in the chocolateyInstall.ps1 script and uninstall if the software is installed. This is most likely an issue with the 'python3' package and not with Chocolatey itself. Please follow up with the package maintainer(s) directly. ERROR: Running ["C:\ProgramData\chocolatey\lib\python3\tools\python-3.10.0-amd64.exe" /quiet InstallAllUsers=1 PrependPath=1 TargetDir="C:\Python310" ] was not successful. Exit code was '1638'. Exit code indicates the following: This MSI requires uninstall prior to installing a different version. Please ask the package maintainer(s) to add a check in the chocolateyInstall.ps1 script and uninstall if the software is installed. This is most likely an issue with the 'python3' package and not with Chocolatey itself. Please follow up with the package maintainer(s) directly.. The install of python3 was NOT successful. Error while running 'C:\ProgramData\chocolatey\lib\python3\tools\chocolateyInstall.ps1'. See log for details.

I had to stop testing after installing a lot of things on the C drive

[wzgrx]

If you want to test the minimal set of files to compare you can checkout branch test_unified_installer

Each reinstallation will result in different errors

[bmaltais]

I never tested the installer on a fresh windows machine... mine had all those pre-req already installed manually... thanks for testing this out.

[wzgrx]

我从未在新的Windows机器上测试过安装程序...我的已经手动安装了所有这些预要求......感谢您对此进行测试。

It's okay, I'm just testing

[bmaltais]

Hello @alicx1 you appear to be using the pre-unified_installer version. Still interesting to know why it is failing.

1st question, did you install the pre-requirements for python 3.10.9 on your computer? I see a Warning: Python version 3.10.9 is recommended.

This might be why things are failing to install...

[alicx1]

@bmaltais sorry wrong info. i just used the wrong python version. i have many installed and it's easy to confuse

[jstayco]

K, I ran an experiment. I applied only a few files from the unified installer on top of the working pre-Unified_installer branch to see if things would work again and they did. Here are the files I copied:

kohya_gui.py
launcher.py
setup.bat
setup.ps1
setup.sh
install_config.yml
requirements_macos.txt

The installer did not properly copy the 4 bitsandbytes files so I had to re-copy them.

Then running the GUI I was able to train with no issue... So the problem must be related to other files that were not supposed to be updated... I feel like replacing all the files in the current pre-unified_installer branch with this minimalist working set of files so we get back to a working state?

The only ones I can think of that would introduce an issue are the .dll files. Just roll those back and see if that issue disappears. All other files are fine. Because the only other changes I made outside the ones listed there are

1. README.md
2. change setup.py to pyproject.toml
3. CONTRIBUTING.md
4. Removed scripts that are no longer needed such as validate_requirements.py, upgrade scripts, gui launcher scripts, etc. Stuff that are launched independently of kohya_gui.py altogether for maintenance purposes that the launcher now handles.

[bmaltais]

I will revert the dev branch to what it was before and copy the files above in there since this appear to be all that is really needed to bring the new unified launcher. THe we can remove unnecessary files from dev as required... but at least the base will be solid

[jstayco]

Rope in my new fixes into the setup.ps1 before you do that, so the enhancements aren't lost. I'm hoping it'll fix false negative situations like what @wzgrx experienced.

[jstayco]

I will revert the dev branch to what it was before and copy the files above in there since this appear to be all that is really needed to bring the new unified launcher. THe we can remove unnecessary files from dev as required... but at least the base will be solid

@bmaltais Make sure the files you include to bring in are:

1. kohya_gui.py
2. launcher.py
3. pyproject.toml
4. setup.bat
5. setup.ps1
6. setup.sh
7. All of the config_files folder including subdirs and files
8. requirements.txt
9. requirements_macos.txt
10. README.md
11. CONTRIBUTING.md

And remove setup.py, so you can bulid project as normal with the new pyproject.toml file.

After you do that and let me know, I'll go back in and remove the extras that can get deleted and change the scripts that you wanted to keep to point at the new stuff.

[bmaltais]

Some of the files that were removed should be kept for windows user legacy gui.bat/ps1.

[jstayco]

Don't worry, just cleaning up things that are now handled in launcher.py and re-aiming gui stuff to go through launcher.py so they get the features too. I think I understand your requirements and it'll be a PR for review of course.

Just make sure you merge that last PR into setup.ps1, then pull the files from dev, so they have all the current PRs merged in. After you rollback and re-merge the files, I'll start the cleanup stuff while testing is being done to make sure it's all good now.

[bmaltais]

OK, while trying to revert all the changes to dev I think I found out the source of the issue with training (the updated kohya_ss.py)... and I apparently did not. Man, I really don't know what is causing all this trouble with training. Something is messing things up but I can't figure out why. Maybe I will just commit the broken solution that sort of work and we fix it without braking training... without functional training there is no point to have an installer if the result won't work for users.

My big challenge is that I already have little time to maintain the GUI... so spending a lot of time figuring the installer issues is killing me. I know it is for the better but the added work is frustrating.

[jstayco]

All I've got for you there is my previous ask:

I'd like to ask if I can get some limited permissions to the repository. My goals are: be able to assign installation-related issues to myself, close those issues as needed, and possibly dev write permissions if you're comfortable with that, so I can push fixes through ASAP since I'm expecting all of the bugs from all kinds of setups.

If you are also comfortable with it, if you give me access to the Github secrets and workflows for this repo, I can start testing out automated Docker builds and publishing capabilities.

Depending on life stuff I'll have a lot of time available for the next 2-4 weeks minimum. I'm having a great time working on this project, so I'm happy to do it end to end.

[bmaltais]

It is just puzzling what make it work and then suddenly it does not. Hard to pin point why it will sometime train in 8bit and other time it just will Traceback.

[bmaltais]

OK, I just pushed a "destructive" commit to the dev branch that will properly train AdamW8bit on Windows as long as I do the following steps:

.\setup.bat
# It will fail after install because I reverted kohya_gui.py to make things work
.\venv\Scripts\activate
python .\tools\update_bitsandbytes.py
python.exe .\kohya_gui.py

Doing thing in that order will properly train.

I will try to upgrade kohya_gui.py to the version that was before my destructuve commit. Usually when I replace it things are falling apart... let test... if things still work I will re-introduce that part.

[bmaltais]

OK, I tested updating kohya_gui.py, redoing the steps above and AdamW8bit still work... so the current dev branch will work except the bitsandbytes files are not properly updated after the pip module installation. I guess I could fix that next.

[bmaltais]

Just as a FYI... Lot's of people are using the GUI. Here is a recent video on how to use it: https://www.youtube.com/watch?v=j-So4VYTL98

We have to be cautious about releasing a very polished update for the setup and retain legacy methods for current users... last thing we should do is pull setup options they are used to and have been shown how to use them... this would create a lot of confusion and frustration.

I know we all know that, but I just wanted to point out the obvious ;-)

[bmaltais]

I fixed (rather crudely because I did not fully understood why the if statement was always false) the issue with bitsandbytes file copy.

[jstayco]

As long as it only triggers on Windows, shouldn't be a big deal.

[jstayco]

print(f"Copying {os.path.join(bitsandbytes_source, 'cextension.py')}")

Don't use print statements anymore. The logging facilities are there, so you can get text logs from users. If you want to always inform the user? logging.critical(msg). But this is just copying files, you want logging.debug(), so you aren't spamming the user during install.

Yeah, just took another look. All of those print statements should be logging.debug("stuff") instead of the print statements. At this point, there should be zero print statements in the code beyond the point we parse arguments. I had to use print statements before the arguments are parsed as that is before the logging system is setup. But those are commented out unless needed for debugging purposes.

These are you print replacements across the board: https://github.com/bmaltais/kohya_ss/blob/dev/CONTRIBUTING.md#logging-functions

[bmaltais]

I added them to make sure I was not missing the statements due to debug levels... they should indeed be switched back to logging.debug now...

I am trying to run the wd14 captionning utility and I run into a weird issue with:

Traceback (most recent call last):
  File "D:\kohya_ss\finetune\tag_images_by_wd14_tagger.py", line 15, in <module>
    import library.train_util as train_util

Not sure why it no longer find the library. I notice the way the library is installed had to change due to setup.sh being now used to setup linux... could it be causing the library issues?

[jstayco]

You should never need to do that. Simply launching it as -vvv or --verbosity 3 will properly print those out with affecting any functionality. And give you text logs too, so you can quickly search them instead of scrolling through hundreds of lines of stuff.

[jstayco]

I notice the way the library is installed had to change due to setup.sh being now used to setup linux

Not sure what you mean here. The kohya_ss library is built the normal way via pip. Only thing that changed there was setup.py -> pyproject.toml. Which should have zero effect and you check debug logs to verify if needed.

Edit: Maybe it's a weird deal because of all the rebuilds? Could try the repair option or just remove venv.

[bmaltais]

Well... wd14 captioning does not work anymore, complaining of import issue... not sure why as nothing changes in kohya_ss code.

Doing pip freeze and I don't see it getting installed... I see kohya_ss but not library... could it be that? Don't remember if it was library or not before... will need to go back to a previous release to test. Possible the name has nothing to do with it too... but just something that sort of jumped at me.

absl-py==1.4.0
accelerate==0.18.0
aiofiles==23.1.0
aiohttp==3.8.4
aiosignal==1.3.1
albumentations==1.3.0
altair==4.2.2
anyio==3.6.2
appdirs==1.4.4
astunparse==1.6.3
async-timeout==4.0.2
attrs==23.1.0
bitsandbytes==0.35.0
cachetools==5.3.0
certifi==2023.5.7
charset-normalizer==3.1.0
click==8.1.3
cmake==3.26.3
colorama==0.4.6
contourpy==1.0.7
cycler==0.11.0
dadaptation==1.5
diffusers==0.10.2
docker-pycreds==0.4.0
easygui==0.98.3
einops==0.6.0
entrypoints==0.4
fairscale==0.4.13
fastapi==0.95.1
ffmpy==0.3.0
filelock==3.12.0
flatbuffers==23.5.8
fonttools==4.39.4
frozenlist==1.3.3
fsspec==2023.5.0
ftfy==6.1.1
gast==0.4.0
gitdb==4.0.10
GitPython==3.1.31
google-auth==2.18.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
gradio==3.28.1
gradio_client==0.2.3
grpcio==1.54.0
h11==0.14.0
h5py==3.8.0
httpcore==0.17.0
httpx==0.24.0
huggingface-hub==0.13.3
idna==3.4
imageio==2.28.1
importlib-metadata==6.6.0
Jinja2==3.1.2
joblib==1.2.0
jsonschema==4.17.3
keras==2.10.0
Keras-Preprocessing==1.1.2
kiwisolver==1.4.4
kohya-ss @ file:///D:/kohya_ss
lazy_loader==0.2
libclang==16.0.0
lightning-utilities==0.8.0
linkify-it-py==2.0.2
lion-pytorch==0.0.6
lycoris-lora==0.1.4
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib==3.7.1
mdit-py-plugins==0.3.3
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.4
mypy-extensions==1.0.0
networkx==3.1
numpy==1.24.3
oauthlib==3.2.2
opencv-python==4.7.0.68
opencv-python-headless==4.7.0.72
opt-einsum==3.3.0
orjson==3.8.12
packaging==23.1
pandas==2.0.1
pathtools==0.1.2
Pillow==9.5.0
protobuf==3.19.6
psutil==5.9.5
pyasn1==0.5.0
pyasn1-modules==0.3.0
pydantic==1.10.7
pydub==0.25.1
pyparsing==3.0.9
pyre-extensions==0.0.23
pyrsistent==0.19.3
python-dateutil==2.8.2
python-multipart==0.0.6
pytorch-lightning==1.9.0
pytz==2023.3
PyWavelets==1.4.1
PyYAML==6.0
qudida==0.0.4
regex==2023.5.5
requests==2.28.2
requests-oauthlib==1.3.1
rsa==4.9
safetensors==0.2.6
scikit-image==0.20.0
scikit-learn==1.2.2
scipy==1.10.1
semantic-version==2.10.0
sentry-sdk==1.22.2
setproctitle==1.3.2
six==1.16.0
smmap==5.0.0
sniffio==1.3.0
starlette==0.26.1
sympy==1.12
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.10.1
tensorflow-estimator==2.10.0
tensorflow-io-gcs-filesystem==0.31.0
termcolor==2.3.0
threadpoolctl==3.1.0
tifffile==2023.4.12
timm==0.6.12
tk==0.1.0
tokenizers==0.13.3
toml==0.10.2
toolz==0.12.0
torch==2.0.0+cu118
torchmetrics==0.11.4
torchvision==0.15.1+cu118
tqdm==4.65.0
transformers==4.26.0
triton @ https://huggingface.co/r4ziel/xformers_pre_built/resolve/main/triton-2.0.0-cp310-cp310-win_amd64.whl#sha256=91a6ec395022743269c942df7af01b210f642fb633d146a811be05a455adbae2        
typing-inspect==0.8.0
typing_extensions==4.5.0
tzdata==2023.3
uc-micro-py==1.0.2
urllib3==1.26.15
uvicorn==0.22.0
voluptuous==0.13.1
wandb==0.15.0
wcwidth==0.2.6
websockets==11.0.3
Werkzeug==2.3.4
wrapt==1.15.0
xformers==0.0.17
yarl==1.9.2
zipp==3.15.0

[jstayco]

Have to run it with debug logs. Because if pip is reporting an error during build, that's where you'll see it. Debug logs or we don't have anything to work with.

[jstayco]

kohya-ss @ file:///D:/kohya_ss

That's the one. Name defined in pyproject.toml.

[bmaltais]

In the master branch there is an entry called

library @ file:///E:/kohya_ss-old

But it is missing in the dev branch. Is there a reason to move away from the original setup.py method? Is it getting deprecated?

[jstayco]

That is correct. Renamed as "library" is just the folder the code is in. Project name is kohya_ss per the pyproject.toml.

[jstayco]

Yes, that was deprecated and throwing a warning every single build in every single mode. Zero reason to stay on old stuff.

[jstayco]

If you want to just do a quick test rename kohya_ss back to library in pyproject.toml :shrug:

[bmaltais]

Still failing:

Traceback (most recent call last):
  File "D:\kohya_ss\finetune\tag_images_by_wd14_tagger.py", line 15, in <module>
    import library.train_util as train_util
ModuleNotFoundError: No module named 'library'

Somehow it can't find the library module