Dependencies resolution problem

[MuhammadAasharibNawshad]

Hello,

This repo donot have freezed dependencies which is causing the dependencies resolution and code running issues. Since many of the the latest packages have updated, ideal solution will be to freeze the dependencies on which video generation is working. Also please specify the python version for smooth dependency resolution.

Thanks,

[AmitMY]

What issues are you encountering? This should work with 3.9 and up

[MuhammadAasharibNawshad]

for running the project i first i create conda env with python 3.9 then i run pip install .

Then I try to run pix2pix and controlnet method using the commands given. but They do not run and give error module not found. for solving this issue I install the required package and then again run the command but again get some other module not found error. this goes on and on...

Further, tensorflow, torch versions are very specific with cuda version. these things also need to be defined as installing latest version of packages are causing cuda incompatibility and other libraries issues.

I think the solution is to freeze the packages version.

[AmitMY]

You should follow the README. For example, if you would like to run pix2pix, you must: pip install '.[pix2pix]'

If you have a specific error, for example you say "module not found", please write THE FULL ERROR here. which module not found? What commands are you running exactly?

[MuhammadAasharibNawshad]

Steps to reproduce: I create a new python 3.9 env. then ran pip install '.[pix2pix]'. After it ran this command

pose_to_video --type=pix2pix --model=pix_to_pix.h5 --pose=assets/testing-reduced.pose --video=assets/outputs/pix2pix.mp4

Logs:

Loading input pose ...
Generating video ...
2024-05-16 09:20:06.097602: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-05-16 09:20:07.040754: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
0it [00:00, ?it/s]
Traceback (most recent call last):
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\aasha\.conda\envs\p_to_v\Scripts\pose_to_video.exe\__main__.py", line 7, in <module>
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\pose_to_video\bin.py", line 63, in main
    for frame in tqdm(frames):
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\tqdm\std.py", line 1181, in __iter__
    for obj in iterable:
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\pose_to_video\conditional\pix2pix\__init__.py", line 40, in pose_to_video
    model = load_model(model_path, compile=False)
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\saving\saving_api.py", line 183, in load_model
    return legacy_h5_format.load_model_from_hdf5(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\legacy\saving\legacy_h5_format.py", line 133, in load_model_from_hdf5
    model = saving_utils.model_from_config(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\legacy\saving\saving_utils.py", line 85, in model_from_config
    return serialization.deserialize_keras_object(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\legacy\saving\serialization.py", line 495, in deserialize_keras_object
    deserialized_obj = cls.from_config(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\models\model.py", line 517, in from_config
    return functional_from_config(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\models\functional.py", line 517, in functional_from_config
    process_layer(layer_data)
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\models\functional.py", line 497, in process_layer
    layer = saving_utils.model_from_config(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\legacy\saving\saving_utils.py", line 85, in model_from_config
    return serialization.deserialize_keras_object(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\legacy\saving\serialization.py", line 495, in deserialize_keras_object
    deserialized_obj = cls.from_config(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\layers\core\wrapper.py", line 43, in from_config
    layer = serialization_lib.deserialize_keras_object(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\saving\serialization_lib.py", line 694, in deserialize_keras_object
    cls = _retrieve_class_or_fn(
  File "C:\Users\aasha\.conda\envs\p_to_v\lib\site-packages\keras\src\saving\serialization_lib.py", line 812, in _retrieve_class_or_fn
    raise TypeError(
TypeError: Could not locate class 'Sequential'. Make sure custom classes are decorated with `@keras.saving.register_keras_serializable()`. Full object config: {'class_name': 'Sequential', 'config': {'name': 'sequential_2', 'layers': [{'class_name': 'InputLayer', 'config': {'batch_input_shape': [None, 256, 256, 3], 'dtype': 'float32', 'sparse': False, 'ragged': False, 'name': 'conv2d_1_input'}}, {'class_name': 'Conv2D', 'config': {'name': 'conv2d_1', 'trainable': True, 'dtype': 'float32', 'filters': 64, 'kernel_size': [4, 4], 'strides': [2, 2], 'padding': 'same', 'data_format': 'channels_last', 'dilation_rate': [1, 1], 'groups': 1, 'activation': 'linear', 'use_bias': False, 'kernel_initializer': {'class_name': 'RandomNormal', 'config': {'mean': 0.0, 'stddev': 0.02, 'seed': None}}, 'bias_initializer': {'class_name': 'Zeros', 'config': {}}, 'kernel_regularizer': None, 'bias_regularizer': None, 'activity_regularizer': None, 'kernel_constraint': None, 'bias_constraint': None}}, {'class_name': 'LeakyReLU', 'config': {'name': 'leaky_re_lu_1', 'trainable': True, 'dtype': 'float32', 'alpha': 0.30000001192092896}}]}}

List of packages installed after pip install '.[pix2pix]'

Package                      Version
---------------------------- -----------
absl-py                      2.1.0
astunparse                   1.6.3
attrs                        23.2.0
certifi                      2024.2.2
cffi                         1.16.0
charset-normalizer           3.3.2
colorama                     0.4.6
contourpy                    1.2.1
cycler                       0.12.1
flatbuffers                  24.3.25
fonttools                    4.51.0
gast                         0.5.4
google-pasta                 0.2.0
grpcio                       1.63.0
h5py                         3.11.0
idna                         3.7
importlib_metadata           7.1.0
importlib_resources          6.4.0
jax                          0.4.28
jaxlib                       0.4.28
keras                        3.3.3
kiwisolver                   1.4.5
libclang                     18.1.1
Markdown                     3.6
markdown-it-py               3.0.0
MarkupSafe                   2.1.5
matplotlib                   3.9.0
mdurl                        0.1.2
mediapipe                    0.10.14
ml-dtypes                    0.3.2
namex                        0.0.8
numpy                        1.26.4
opencv-contrib-python        4.9.0.80
opencv-python                4.9.0.80
opt-einsum                   3.3.0
optree                       0.11.0
packaging                    24.0
pillow                       10.3.0
pip                          24.0
pose_format                  0.4.1
pose-to-video                0.0.1
protobuf                     4.25.3
pycparser                    2.22
Pygments                     2.18.0
pyparsing                    3.1.2
python-dateutil              2.9.0.post0
requests                     2.31.0
rich                         13.7.1
scipy                        1.13.0
setuptools                   69.5.1
six                          1.16.0
sounddevice                  0.4.6
tensorboard                  2.16.2
tensorboard-data-server      0.7.2
tensorflow                   2.16.1
tensorflow-intel             2.16.1
tensorflow-io-gcs-filesystem 0.31.0
termcolor                    2.4.0
tqdm                         4.66.4
typing_extensions            4.11.0
urllib3                      2.2.1
Werkzeug                     3.0.3
wheel                        0.43.0
wrapt                        1.16.0
zipp                         3.18.1

[AmitMY]

Following your commands, it works just fine. (Please see how I edited your comment. formatting text helps readability, please do so next time)

See my reproduction: https://colab.research.google.com/drive/1HS-h3jI78jMhU6hR27R1cMdR5rb_gmoC?usp=sharing

[sparkkid1234]

hi @AmitMY firstly thank you for open-sourcing this, this is pretty incredible. I did encounter the same exact code breaking dependencies resolution problem @MuhammadAasharibNawshad encountered.

I did manage to fix it and am relaying my findings here for future reference. The main problem is with the keras model loading. It seems like the models were trained using tensorflow 2.10.0 (based on the docker image) which uses keras 2.x.x while the latest tensorflow 2.16.0 uses keras 3.x.x which leads to the code breaking when trying to load the model. @MuhammadAasharibNawshad you can fix this by using any version of tensorflow with keras 2.x.x, so far I can confirm TF 2.10.0 -> 2.15.0 all works.

[AmitMY]

Thanks for the information @sparkkid1234 !

[Tortoise17]

@AmitMY tensorflow current release 2,16,x has no available packages. while downgrade to 2,14,x also doesn't provide the proper export of results.

[Tortoise17]

@AmitMY Finally able to run. It works properly. Thankyou for being helpful always.