Closed OrangeSodahub closed 10 months ago
I am unable to reproduce this.
Could you ensure:
Yes, I'm sure, it's so wierd and I'm seeking more possible reasons... While I found that the true stuck point is the make_train_dataset
, while I'm using my customed dataset.
I'm so confused how to correctly load my own dataset, your instruction https://huggingface.co/docs/datasets/v2.0.0/en/dataset_script here seems not consistent with the training script train_controalnet.py
, do I need to replace the load_dataset
function with mine? And according to this instruction, I create a new datasete class, but no new load_dataset
func, do I needd to create a new one? THANKS, I'm so confused.
Here if I sepcify the local dataset dir, I didn't find any instructions about how I need to organize my dataset direction, what's more, if I want to use a new dataset class, no instructions about how to connect load_dataset
and my new dataset class.
https://github.com/huggingface/diffusers/blob/9135e54e768a59ddcf8ad18818d2ffe69ea3a32a/examples/controlnet/train_controlnet.py#L596-L599
cc @sayakpaul I just wonder how to organize my local dataset so as to use load_dataset
in train_controlnet.py
Cc: @lhoestq for suggesting ways to organize the dataset locally to mimic ControlNet training.
Quentin, the online dataset reference is here: https://huggingface.co/datasets/fusing/fill50k
You need a dataset with three columns:
image
of type Imageconditioning_image
of type Imagetext
of type stringTo have a dataset with these columns you can for example have a local directory with the same structure as fill50k
.
We are also seeing if we can have an easy structure, like
train/metadata.csv
train/image0.png
train/image1.png
...
train/conditioning_image0.png
train/conditioning_image1.png
...
but so far the metadata.csv
can only link to one image per example.
See this issue for more information: https://github.com/huggingface/datasets/issues/5760
@lhoestq Thanks for your explaination. I agree with that it's defintely doable if I organize the local dataset totally same as the fill50k
, but I want to know more about the deeper logic that your load_dataset
process a local dataset folder, so that I could adjust my data according to my thinkings, what is, maybe there are multiple conditons per image.
All I want to do is to do like with the torch's dataset. Your load_dataset
provided the local folder api but it's troublesome to findout the instructions.
The docs are here: https://huggingface.co/docs/datasets/image_dataset.
Then about the controlnet training in particular, I found the docs about the dataset here but imo we can add a mention to this
You need a dataset with three columns:
image
of type Imageconditioning_image
of type Imagetext
of type string
and maybe redirect to the datasets
docs on how to create an image dataset
Thanks, the first docs are useful to me
Hi @OrangeSodahub
is your issue resolved?
YiYi
@yiyixuxu Not yet, I haven't had a chance to check I will add new comments if I have other problems
You should pass the path of the file in the repo instead of the URL :)
@lhoestq After debugging I know need to directly assign the data_dir
in split_dataset
, however the unclear docs make it troublesome
@lhoestq Here it can only fetch the dataset through url right? It doesn't provide the local data dir parameter.
def _split_generators(self, dl_manager):
archive_path = dl_manager.download(_BASE_URL)
split_metadata_paths = dl_manager.download(_METADATA_URLS)
return [
datasets.SplitGenerator(
name=datasets.Split.TRAIN,
gen_kwargs={
"images": dl_manager.iter_archive(archive_path),
"metadata_path": split_metadata_paths["train"],
},
),
datasets.SplitGenerator(
name=datasets.Split.VALIDATION,
gen_kwargs={
"images": dl_manager.iter_archive(archive_path),
"metadata_path": split_metadata_paths["test"],
},
),
]
You can pass the paths to the dl_manager.download function
@lhoestq Thanks. But I load the jsonl
file through load_dataset
, and the two funcs split_generators
and generate_examples
seem not be called at all, and for the script below:
def preprocess_train(examples):
print(examples)
image_transforms = transforms.Compose(
[
transforms.Resize(512, interpolation=transforms.InterpolationMode.BILINEAR),
transforms.CenterCrop(512),
transforms.ToTensor(),
transforms.Normalize([0.5], [0.5]),
]
)
images = [image.convert("RGB") for image in examples["target"]]
images = [image_transforms(image) for image in images]
examples["pixel_values"] = images
return examples
def main():
dataset = load_dataset("./data/") # load jsonl files
train_dataset = dataset["train"]
print(len(train_dataset))
print(train_dataset.column_names)
# Set the training transforms
train_dataset = dataset["train"].with_transform(preprocess_train)
train_dataloader = torch.utils.data.DataLoader(train_dataset, shuffle=True, batch_size=1, num_workers=4)
for data in train_dataloader:
print(data)
if __name__ == "__main__":
multiprocessing.freeze_support()
main()
the example
seems not load the image correctly, it still is a string. Each example
is a line in jsonl
file, is that right? I need to load the image mannually?
Make sure the dataset script file has a correct filename. It must be the same as the dataset repository/folder name to be used
@lhoestq You mean the dataset folder must be at the same level as the new load dataset script? e.g.:
-- ...
|- mydataset.py
|- data
|- train.jsonl
|- val.jsonl
|- test.jsonl
|- train
|- ...
But if I want to load data from another place, how should I do?
And, the key of image filename is fixed? I use target
and source
as my image paths' key to be loaded in jsonl
file, if it doesn't work, how could I successfully uses my keys?
This issue is becoming more of a data loading issue and seems to better off elsewhere. Do you mind continuing the discussion on the datasets
repo instead?
@sayakpaul @patrickvonplaten , i want to train controlnet on stable diffusion inpaint, Can you please tell me how to copy the weights.
Describe the bug
I use the given script to train controlnet however it stuck in initializing weights from unet for a long long time
And I watch the vram and notice that it load the weights again and again, e.g. memory used from 6.5GB to 7.6GB, and then fell down to 6.5GB, and rised to 7.6GB again, and keep repeating ... while the total memory > 100GB
Reproduction
Logs
System Info
VRAM > 100GB, GPU memory 24GB.
Who can help?
Hope some useful solutions! I'm new to this awesome thing. @sayakpaul @patrickvonplaten @yiyixuxu @DN6