search

fjxmlzn / DoppelGANger

[IMC 2020 (Best Paper Finalist)] Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions
http://arxiv.org/abs/1909.13403
BSD 3-Clause Clear License
296 stars 75 forks source link

Training time #43

Closed Xingyyy01 closed 1 year ago

Xingyyy01 commented 1 year ago

hii, first of all, thanks for your great work! I was able to run the example codes, but i got a problem that as I run the code multiple times, the time of per epoch multiplies. Maybe I didn't understand the code enough, trying to get some help.

fjxmlzn commented 1 year ago

What do you mean by "the time of per epoch multiplies"? If you don't mind, you can give me the exact time that you are seeing, and the server setup (e.g., what GPU you are using)

Xingyyy01 commented 1 year ago

Thanks for your reply, i use NVIDIA TITAN Xp. The time of per epoch multiplies: when I run the main.py in example_training for the first time, it takes about 2 minutes per epoch, when I run the file for the second time, it takes about 4 minutes per epoch, when I run it for the third time, it takes about 14 minutes per epoch, and when I run it for the last time, it takes 40 minutes per epoch. Attachments are files produced by running code. time1.txt time2.txt time3.txt

fjxmlzn commented 1 year ago

How many concurrent training jobs are you running on the GPU? you can use nvidia-smi to see them. If you are running multiple jobs are the same time, the training time would increase

Xingyyy01 commented 1 year ago

ooh! I see, thank you! I haved reconfigured the environment these days, and it can run normally, it takes about 6 minutes for one epoch Here is the work.log file generated by the current run, could you help me check it? worker.log

fjxmlzn commented 1 year ago

It looks normal to me!

Xingyyy01 commented 1 year ago

great! tysm!