Closed NotNANtoN closed 2 years ago
Hello,
I see many issues with the code posted above:
But the most problematic thing is class SubsampleSents(torch.nn.Module)
. It is a pre-processing function written in python (=extremely slow) that is single core (as many times slower as you have number of cores on your machine) and runs in the forward pass of your model. Without profiling your code I can tell that's where at least 80% of the time is lost (and because this is single threaded code I'm sure you are not even using a lot of your CPU resources, let alone your GPU).
Hi, thanks a lot for the swift response and for looking into the code!
Referring to your points raised:
if not os.path.exists(path)
non_blocking=True
should be better. I now changed that.NormalizeImage
function and I'm using it now. Side note: It confuses me a bit that NormalizeImage seems to operate on tensors and yet takes numpy arrays as a dtype for mean and std and for the type
argument.The point about the slow SubsampleSents
- I fully agree, the problem is that I had quite some pain in trying to implement it into the pipeline. I now understand that there always needs to be a np.array or tensor in the pipeline, before I did output some text. So I made a new transform that does the 3 previous transforms in one - I had to set the jit_mode
to False, otherwise it would not train:
class UintArrToToken(Operation):
def __init__(self, sent_frac):
self.sent_frac = sent_frac
# Return the code to run this operation
def generate_code(self):
tokenize = Tokenizer()
def uintarr_to_text(x):
return x.tobytes().decode('ascii').strip()
if self.sent_frac > 0:
subsample = SubsampleSentsNp(self.sent_frac)
def uint_to_token(x, dst):
text = uintarr_to_text(x)
text = subsample(text)
tokens = tokenize(text)
return tokens
else:
def uint_to_token(x, dst):
text = uintarr_to_text(x)
tokens = tokenize(text)
return tokens
return uint_to_token
def declare_state_and_memory(self, previous_state):
new_state = replace(previous_state, shape=(77,), dtype=np.int32, jit_mode=False)
mem_allocation = AllocationQuery((77,), np.int32)
return (new_state, mem_allocation)
class SubsampleSentsNp: def init(self, frac, min_sent_len=5): super().init() self.frac = frac self.min_sent_len = min_sent_len
def __call__(self, x):
sents = [sent.strip() for sent in x.split(".") if len(sent) > self.min_sent_len]
num_sents_used = round(len(sents) * self.frac)
sents_used = np.random.choice(sents, num_sents_used, replace=False)
text = ". ".join(sents_used)
return text
and my pipelines are now:
base_pipeline = [
FFCVT.ToTensor(),
FFCVT.ToDevice(device, non_blocking=True),
FFCVT.ToTorchImage(),
FFCVT.NormalizeImage(mean, std, np.float16),
FFCVT.Convert(torch.float16),
FFCVT.ToDevice(device, non_blocking=True),
]
self.train_image_pipeline = [
RandomResizedCropRGBImageDecoder((224, 224)),
FFCVT.Cutout(8, tuple(map(lambda x: int(x * 255), mean))),
FFCVT.RandomTranslate(padding=10, fill=mean),
#TF.ColorJitter(brightness=0.1, contrast=0.1, saturation=0.1, hue=0.0),
] + base_pipeline
self.test_image_pipeline = [ CenterCropRGBImageDecoder((224, 224), 1), ] + base_pipeline
self.labels_pipeline = [NDArrayDecoder(), FFCVT.ToTensor(), FFCVT.ToDevice(device, non_blocking=True),]
self.train_text_pipeline = [NDArrayDecoder(),
#SubsampleSents(self.sent_frac),
#Tokenize(),
UintArrToToken(self.sent_frac),
FFCVT.ToTensor(),
FFCVT.ToDevice(device, non_blocking=True),]
The good thing: it is now faster than before! With FFCV it runs at 4.38 it/s and without at 2.26 it/s. So thanks! But the bad things: 1. I think this is not optimal, especially considering that I don't use the `jit`and that I cannot use the `ColorJitter`. 2. Even worse: somehow the labels seem mixed up when I load them - for some classes I now have 0 positive examples in the labels. I'm not sure if you can see how this could happen?
Btw, when I try to uncomment the TF.ColorJitter
in my pipeline I get the following error. Any clues there? Do I need to write a custom FFCV transform even for the torchvision transforms? Should I make a separate issue?:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/srv/home/8wiehe/miniconda3/envs/ffcv/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/srv/home/8wiehe/miniconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/loader/epoch_iterator.py", line 79, in run
result = self.run_pipeline(b_ix, ixes, slot, events[slot])
File "/srv/home/8wiehe/miniconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/loader/epoch_iterator.py", line 133, in run_pipeline
result = code(*args)
File "/srv/home/8wiehe/miniconda3/envs/ffcv/lib/python3.9/site-packages/numba/core/dispatcher.py", line 468, in _compile_for_args
error_rewrite(e, 'typing')
File "/srv/home/8wiehe/miniconda3/envs/ffcv/lib/python3.9/site-packages/numba/core/dispatcher.py", line 409, in error_rewrite
raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name 'self': Cannot determine Numba type of <class 'ffcv.transforms.module.ModuleWrapper'>
File "../miniconda3/envs/ffcv/lib/python3.9/site-packages/ffcv/transforms/module.py", line 25:
def apply_module(inp, _):
res = self.module(inp)
^
During: resolving callee type: type(CPUDispatcher(<function ModuleWrapper.generate_code.<locals>.apply_module at 0x7f593b13cd30>))
During: typing of call at (2)
During: resolving callee type: type(CPUDispatcher(<function ModuleWrapper.generate_code.<locals>.apply_module at 0x7f593b13cd30>))
During: typing of call at (2)
File "/srv/home/8wiehe/master_thesis", line 2:
<source missing, REPL/exec in use?>
So there is one thing I don't understand about FFCV and dataloaders in general: how can it be a bottleneck? If we use 16 workers for a batch size of 16, then each worker only needs to load one image, apply some minor transforms on it, synchronize them in a batch and send the batch to the GPU. So it can only be a bottleneck if these operations take longer than the forward+backpass, right?
Anyways, I think I'm giving up. I tried working with the text, tokenized in the beton file, but in the end my speed-up is minor and the training runs, but the results are gibberish and my labels mixed up. Which sucks because I spent quite some days trying to get this to work. Thanks anyways for your feedback!
Many reasons why it is a bottleneck:
Basically in this situation you are limiting yourself to a single worker + run python code instead which is orders of magnitude slower than compiled code.
Hope it helps clarifying!
Closing due to inactivity
Hi dear FFCV team,
I guess I am trying something that is not yet fully supported:
But for both problems there seem to be solutions.
custom_PTL_methods.py
So, I am glad that I got FFCV running. For reference, these are my relevant code snippets: Creating the .beton files:
The creation of the actual .beton files and of the pipelines (I lef some transforms there that I commented out as it did not work with them included in the pipeline):
This is my dataloader:
The last step where the batch is actually used in train step, along with the relevant transforms:
The results are quite bad, unfortunately. It takes 45 hours with FFCV and without only 28 hours. Proud-sunset is the FFCV run:![image](https://user-images.githubusercontent.com/19983153/153858945-f5a64027-99bb-4f82-bc4e-648a7398a271.png)
Also, the performance is worse (maybe related due to the ColorJitter missing now?). A higher zero-shot validation accuracy is better:![image](https://user-images.githubusercontent.com/19983153/153859109-230fb3de-16c5-48e4-9d86-29229d0d9892.png)
So I know that I left some transforms out from the pipeline (like the FFCVT.ToDevice(device, non_blocking=True), which might be important) but it simply did not run with them in there and the error messages were completely cryptic. I also think that having the sentence subsampling and tokenization in the
train_step
is far from ideal.I would be happy about any feedback how I could improve this to actually give me a speed-up!