Raise ValueError when running multi_t2t mode

[nvidiafucku]

Following README.md, I ran the code below:

Text->Projection on Planck scene: python train.py --config configs/planck.txt --render_only --render_modes multi_t2t --cdc_conda /home/admin/local/mambaforge/envs/ldm --cdc_src /home/admin/projects/cdc/src

But this was terminated with following traceback:

INFO:root:Reloading from: /mnt/data/admin/nepmap/release/logs/planck/latest.tar
INFO:root:sandbox mode: multi_t2t
0: Side profile of The Batman
Traceback (most recent call last):
  File "/home/admin/projects/nepmap/nepmap-server/train.py", line 277, in <module>
    render_sandbox(radiance_field, occupancy_grid, scene_aabb,
  File "/home/admin/projects/nepmap/nepmap-server/helpers.py", line 715, in render_sandbox
    CDC([extra_info["prompt"][i]], image,
  File "/home/admin/projects/nepmap/nepmap-server/helpers.py", line 462, in CDC
    result = gsoup.load_images(output_files, to_float=True, to_torch=True, device=src_image.device)
  File "/home/admin/local/mambaforge/envs/nepmap1/lib/python3.9/site-packages/gsoup/gsoup_io.py", line 142, in load_images
    images = np.stack(images, axis=0)
  File "/home/admin/local/mambaforge/envs/nepmap1/lib/python3.9/site-packages/numpy/core/shape_base.py", line 445, in stack
    raise ValueError('need at least one array to stack')
ValueError: need at least one array to stack

The version of the packages I use:

• python: 3.9.18

[yoterel]

Hi there, This does not reproduce on my end. One guess I have is that the reference to the cdc conda environment isn't working. Can you please let me know how do you activate that environment? (i.e. "conda activate /home/admin/local/mambaforge/envs/ldm" ? or is it a named environment ?) If it is named, can you try replacing: --cdc_conda /home/admin/local/mambaforge/envs/ldm with: --cdc_conda name_of_your_environment

Also can you make sure that the conda environment exists at "/home/admin/local/mambaforge/envs/ldm"? Please also let me know of your OS version?

[nvidiafucku]

Hi there, This does not reproduce on my end. One guess I have is that the reference to the cdc conda environment isn't working. Can you please let me know how do you activate that environment? (i.e. "conda activate /home/admin/local/mambaforge/envs/ldm" ? or is it a named environment ?) If it is named, can you try replacing: --cdc_conda /home/admin/local/mambaforge/envs/ldm with: --cdc_conda name_of_your_environment

Also can you make sure that the conda environment exists at "/home/admin/local/mambaforge/envs/ldm"? Please also let me know of your OS version?

Thanks, I use mamba instead of conda. I activate the environment with "mamba activate ldm". I tried the replacement above, and got the same info.

I'm sure that the conda environment exists at "/home/admin/local/mambaforge/envs/ldm". Is there any possible I incorrectly installed it?

My OS is: Ubuntu 20.04.6 LTS

[yoterel]

I see, then you will have to manually edit the function CDC.

you probably just need to edit the command line arguments that are built here, i.e.:

I think you probably just need to edit the command line string and replace conda run with mamba run (disclaimer: I have never used mamba before).

Another thing that might fix it if the above doesn't help, is to replace --prefix {1} with --name {1} and supply the CDC mamba environment name instead of full path.

[nvidiafucku]

Thanks,

I checked the function CDC and found that the problem was inside the subprocess, which led me to overlook the problem inside the subprocess because the stdout and stderr of the subprocess were not printed out.

Then I solved some problems to make img2img_inpaint.py work fine when run on its own.

I added a few lines of code to output the stream of the subprocess:

stdout, stderr = subprocess.Popen(f"{cmd}", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
stdout_text = stdout.decode('utf-8')
stderr_text = stderr.decode('utf-8')
print("Standard Output:", stdout_text)
print("Standard Error:", stderr_text)

And I ran the command of nepmap agagin, then I got:

img2img_inpaint.py: error: ambiguous option: --strength could match --strength_in, --strength_out

[nvidiafucku]

I simply deleted '--strength 1', and I got these: ( Some of the similarly extensive content was omitted by me using symbols '...')

INFO:root:loading: 296 / 297
INFO:root:Reloading from: /mnt/data/admin/nepmap/release/logs/planck/latest.tar
INFO:root:sandbox mode: multi_t2t
0: Side profile of The Batman
Standard Output: Loading model from models/ldm/stable-diffusion-v1/sd-v1-5-inpainting.ckpt
Global Step: 440000
LatentInpaintDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.54 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Sweeping through: {'T_in': [0.0, 0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5]}
Total of 9 experiments
===================================================
Running with (('T_in', 0.0),)
reading images from /mnt/data/adminnepmap/release/my_logs/logs_20231113/planck/render_only/multi_t2t/diffuse_0/tmp_input/images/0000.png
reading masks from /mnt/data/admin/nepmap/release/my_logs/logs_20231113/planck/render_only/multi_t2t/diffuse_0/tmp_input/masks/0000.png
Running DDIM Sampling with 50 timesteps
finished 1/9 experiments
===================================================
Running with (('T_in', 0.005),)
reading images from /mnt/data/admin/nepmap/release/my_logs/logs_20231113/planck/render_only/multi_t2t/diffuse_0/tmp_input/images/0000.png
reading masks from /mnt/data/admin/nepmap/release/my_logs/logs_20231113/planck/render_only/multi_t2t/diffuse_0/tmp_input/masks/0000.png
Running DDIM Sampling with 50 timesteps
finished 2/9 experiments

**...**

===================================================
Running with (('T_in', 0.5),)
reading images from /mnt/data/admin/nepmap/release/my_logs/logs_20231113/planck/render_only/multi_t2t/diffuse_0/tmp_input/images/0000.png
reading masks from /mnt/data/admin/nepmap/release/my_logs/logs_20231113/planck/render_only/multi_t2t/diffuse_0/tmp_input/masks/0000.png
Running DDIM Sampling with 50 timesteps
finished 9/9 experiments
Your samples are ready and waiting for you here:
/mnt/data/admin/nepmap/release/my_logs/logs_20231113/planck/render_only/multi_t2t/diffuse_0/tmp_output/00

Enjoy.

Standard Error: Some weights of the model checkpoint at /home/admin/.cache/huggingface/transformers were not used when initializing CLIPTextModel: ['vision_model.encoder.layers.15.mlp.fc2.weight', 'vision_model.encoder.layers.13.mlp.fc1.weight', 
**...** 
- This IS expected if you are initializing CLIPTextModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing CLIPTextModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Data: 0it [00:00, ?it/s]Global seed set to 42
Decoding image: 100%|██████████| 50/50 [00:04<00:00, 11.39it/s]
Sampling: 100%|██████████| 1/1 [00:04<00:00,  4.48s/it]
Data: 1it [00:04,  4.90s/it]/1 [00:04<00:00,  4.48s/it].57it/s]
Data: 0it [00:00, ?it/s]Global seed set to 42

**...**

Data: 0it [00:00, ?it/s]Global seed set to 42
Decoding image: 100%|██████████| 50/50 [00:04<00:00, 11.30it/s]
Sampling: 100%|██████████| 1/1 [00:04<00:00,  4.52s/it]
Data: 1it [00:04,  4.59s/it]/1 [00:04<00:00,  4.52s/it].46it/s]

Traceback (most recent call last):
  File "/home/admin/projects/nepmap/nepmap-server/train.py", line 277, in <module>
    render_sandbox(radiance_field, occupancy_grid, scene_aabb,
  File "/home/admin/projects/nepmap/nepmap-server/helpers.py", line 720, in render_sandbox
    CDC([extra_info["prompt"][i]], image,
  File "/home/admin/projects/nepmap/nepmap-server/helpers.py", line 467, in CDC
    result = gsoup.load_images(output_files, float=True, to_torch=True, device=src_image.device)
  File "/home/admin/local/mambaforge/envs/nepmap1/lib/python3.9/site-packages/gsoup/gsoup_io.py", line 229, in load_images
    images = np.stack(images, axis=0)
  File "/home/admin/local/mambaforge/envs/nepmap1/lib/python3.9/site-packages/numpy/core/shape_base.py", line 445, in stack
    raise ValueError('need at least one array to stack')
ValueError: need at least one array to stack

[yoterel]

Thanks ! I'll try my best to help with this.

Looks like CDC generation succeeded, but finding the output files failed for some reason (i.e. looks like the function "load_images" failed, but output_files seems to point at the right place).

1) Can you confirm that this location /mnt/data/admin/nepmap/release/my_logs/logs_20231113/planck/render_only/multi_t2t/diffuse_0/tmp_output/00 has multiple folders inside, each containing .png files ? (i.e. results from CDC) if so, what are the file names?

2) Since you mentioned that --strength 1 had to be removed, this suggests CDC's instructions no longer work properly, as the underlying diffusers library changed their API. This might suggest the generated files are named differently. I will try installing CDC from scratch and see if I can debug this as well.

[nvidiafucku]

Thanks,

Each subfolder contains 0000_00.jpg, grid-0000.jpg, and config.yaml. I think that should be the problem. I modified output_files to .jpg, and the program is now running.

[yoterel]

Awesome, please share some results if you try a new scene / prompts :)

And if time permits, submit a pull request with the fixes...