default_collate error

[Ishihara-Masabumi]

When I tried to train imagen with the command as follows,

python3 imagen.py --train --source ./holo --imagen model.pth --device cpu

the default_collate error occurred, as follows.

For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on t5-large automatically truncating your input to 512 when padding/encoding.
- If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.
  warnings.warn(
Epoch 1:   6%|▌         | 2/33 [02:31<32:36, 63.11s/batches, epoch_loss=1.37, loss=1.33]   Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.
('1girl, absurdres, animal_ears, bangs, blue_shirt, blue_sky, brown_hair, brown_skirt, building, cloud, commentary_request, day, field, grass, highres, holo, letterboxed, long_hair, nature, outdoors, rock, rural, scenery, shirt, skirt, sky, solo, spice_and_wolf, standing, suzu_(susan_slr97), tree, wide_shot, wolf_ears', '1girl, :d, alternate_breast_size, animal_ears, animated, apple, barefoot, breasts, fang, fang_out, feet, feet_up, food, fruit, holo, large_breasts, long_hair, looking_at_viewer, looping_animation, lying, on_stomach, open_mouth, orange_hair, patreon_username, red_eyes, rtil, smile, soles, solo, spice_and_wolf, swinging_legs, tail, tail_wagging, the_pose, toes, ugoira, watermark, web_address, wolf_ears, wolf_girl, wolf_tail', None, '2girls, animal_ear_fluff, animal_ears, animal_penis, barefoot, breasts, brown_hair, collarbone, dog_penis, ear_sex, erection, feet, futa_with_futa, futanari, highres, holo, incest, knotted_penis, long_hair, multiple_girls, myuri_(spice_and_wolf), navel, nipples, open_mouth, penis, precum, precum_drip, red_eyes, rolling_eyes, seraziel, simple_background, sitting, skull_fucking, small_breasts, solo_focus, spice_and_wolf, standing, tail, testicles, thighhighs, uncensored, white_background, wolf_ears, wolf_girl, wolf_tail', '1girl, animal_ear_fluff, animal_ears, collarbone, dress, flat_chest, floating_hair, greyscale, hand_on_own_cheek, hand_on_own_face, head_tilt, holo, long_hair, monochrome, open_mouth, sketch, smile, solo, spice_and_wolf, tail, tsuchida_ryou, white_background, wolf_ears, wolf_girl, wolf_tail', '1girl, animal_ears, anklet, breasts, brown_hair, capri_pants, cloud, commentary, english_commentary, highres, holo, jewelry, long_hair, lying, nail_polish, no_socks, pants, pouch, red_eyes, seraziel, shoes, signature, sky, small_breasts, solo, spice_and_wolf, star_(sky), tail, tail_in_mouth, wagon, white_nails, wolf_ears, wolf_girl, wolf_tail', '1boy, 1girl, blonde_hair, blue_eyes, blue_sky, blush, brown_hair, cloud, cloudy_sky, fermi_amarti, holo, lipstick, makeup, non-web_source, outdoors, red_eyes, sky, smile, spice_and_wolf, third-party_edit, wolf_girl', '2girls, animal_ears, artist_name, barefoot, blonde_hair, blush, breasts, brown_hair, commentary, commission, completely_nude, cross-section, crossover, doggystyle, english_commentary, fox_ears, fox_girl, fox_tail, futa_with_female, futanari, hair_between_eyes, highres, holo, long_hair, multiple_girls, navel, nipples, nude, open_mouth, penis, senko_(sewayaki_kitsune_no_senko-san), seraziel, sewayaki_kitsune_no_senko-san, sex, sex_from_behind, sheet_grab, simple_background, small_breasts, spice_and_wolf, tail, tail_grab, tongue, tongue_out, uncensored, white_background, wolf_ears, wolf_girl, wolf_tail, yellow_eyes')
Epoch 1:   9%|▉         | 3/33 [02:31<17:13, 34.47s/batches, epoch_loss=1.36, loss=1.33]Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.
('1girl, animal_ear_fluff, animal_ears, barefoot, breasts, brown_hair, crunchobar, fang, feet, hair_over_breasts, highres, holo, knees_to_chest, knees_up, long_hair, looking_at_viewer, nude, open_mouth, outdoors, petite, red_eyes, sitting, small_breasts, smile, solo, spice_and_wolf, tail, tail_censor, thighs, toes, wolf_ears, wolf_girl, wolf_tail', '1girl, :d, all_fours, animal_ear_fluff, animal_ears, arm_support, artist_name, bangs, barefoot, border, breasts, cleavage, collarbone, dress, hair_between_eyes, highres, holo, looking_at_viewer, off-shoulder_dress, off_shoulder, open_mouth, outside_border, patreon_username, pouch, purple_dress, red_eyes, short_dress, side_slit, sleeves_past_wrists, small_breasts, smile, solo, spice_and_wolf, tail, white_border, wolf_ears, wolf_tail, zasshu', '1girl, animal_ear_fluff, animal_ears, animal_penis, artist_name, bangs, bent_over, blunt_bangs, breasts, brown_hair, candle, closed_mouth, commentary, dog_penis, english_commentary, erection, futanari, halloween, happy_halloween, highres, holo, horns, long_hair, looking_at_viewer, nipple_piercing, nipples, penis, piercing, precum, pumpkin, red_eyes, seraziel, small_breasts, solo, spice_and_wolf, tail, testicles, tongue, tongue_out, topless, uncensored, wolf_ears, wolf_girl, wolf_tail', '1girl, akashiro_yulice, animal_ear_fluff, animal_ears, bodypaint, breasts, brown_hair, commentary, commission, cup, disposable_cup, english_commentary, exhibitionism, highres, holding, holding_cup, holding_marker, holo, iced_latte_with_breast_milk_(meme), indoors, long_hair, marker, meme, naked_paint, navel, nipples, painted_clothes, public_indecency, public_nudity, red_eyes, small_breasts, solo, spice_and_wolf, standing, starbucks, tail, wolf_ears, wolf_girl, wolf_tail', "6+boys, 6+girls, amarty_fermi, animal_ears, anniversary, ayakura_juu, black_hair, blonde_hair, border_collie, brown_hair, coin, commentary_request, craft_lawrence, dian_rubens, dog, elsa_schtingheim, enekk, eu_landt, eve_bolan, everyone, facial_hair, fang, fur_trim, goatee, helena_(spice_and_wolf), highres, holding_hands, holding_person, holo, hugging_own_legs, jacket, long_hair, long_sleeves, marc_cole, multiple_boys, multiple_girls, myuri_(spice_and_wolf), nora_arento, official_art, one_eye_closed, open_mouth, outstretched_arms, pouch, priestess, red_eyes, richten_marlheit, shepherd, shepherd's_crook, short_hair, skirt, sky, smile, spice_and_wolf, spread_arms, tail, tankard, tote_col, weiz, wolf_ears, wolf_girl, wolf_tail, zheren", None, '1girl, animal_ears, artist_name, autumn, autumn_leaves, barefoot, blush, brown_hair, coffee, coffee_mug, commission, cup, fern, highres, holo, leg_up, long_hair, mug, nyaxxy, pajamas, red_eyes, smile, solo, spice_and_wolf, tail, window, wolf_ears, wolf_tail', None)
Epoch 1:  12%|█▏        | 4/33 [02:32<18:22, 38.03s/batches, epoch_loss=1.35, loss=1.33]
Traceback (most recent call last):
  File "/home/****/PycharmProjects/imagen/deep_imagen/imagen.py", line 801, in <module>
    main()
  File "/home/****/PycharmProjects/imagen/deep_imagen/imagen.py", line 161, in main
    train(args)
  File "/home/****/PycharmProjects/imagen/deep_imagen/imagen.py", line 545, in train
    for data in tepoch:
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/home/i****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1356, in _next_data
    return self._process_data(data)
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
TypeError: Caught TypeError in DataLoader worker process 4.
Original Traceback (most recent call last):
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 175, in default_collate
    return [default_collate(samples) for samples in transposed]  # Backwards compatibility.
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 175, in <listcomp>
    return [default_collate(samples) for samples in transposed]  # Backwards compatibility.
  File "/home/****/PycharmProjects/imagen/venv/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 183, in default_collate
    raise TypeError(default_collate_err_msg_format.format(elem_type))
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'NoneType'>

I have no idea. Please let me know how to fix it.

[deepglugs]

what do your tag files look like?

[Ishihara-Masabumi]

The contents of one of my tag files are as follows:

1girl, animal_ears, bangs, bracelet, brown_hair, brown_shirt, brown_skirt, closed_mouth, collarbone, commentary, commission, english_commentary, floating_hair, grey_background, hand_on_hip, holo, jewelry, layered_sleeves, long_hair, long_sleeves, looking_at_viewer, red_eyes, rheamii, shirt, short_over_long_sleeves, short_sleeves, simple_background, skirt, smile, solo, spice_and_wolf, tail, very_long_hair, wolf_ears, wolf_tail

It is the one of holo datesets.

[deepglugs]

Hmm. I just pushed an update, do a git pull and see if the problem can be reproduced.

[Ishihara-Masabumi]

Thank you! The training using new datasets seem to be good.

BTW, this training seems to be two parts, training step and sampling time step. Please let me know what exactly imagen is learning at each step.

[deepglugs]

during each training step, I think it's just learning how to denoise. During sampling, it's doing this process over and over again.

[Ishihara-Masabumi]

Thanks a lot.