libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)
https://ffcv.io
Apache License 2.0
2.87k stars 179 forks source link

Google Colab Error with CIFAR-10 example: "ValueError: The number of threads must be between 1 and 2" #382

Open AntonioMacaronio opened 4 months ago

AntonioMacaronio commented 4 months ago

I'm trying to follow the example of FFCV on CIFAR-10 here, except I'm just copy-pasting the code onto google colab. Here is the cell I'm running (it's just copy paste from the docs):

image

Here is the error I ran into: (I'm connected to a T4 runtime)


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-16-1b60a993f99a>](https://localhost:8080/#) in <cell line: 8>()
     26 
     27     # Create loaders
---> 28     loaders[name] = Loader(f'/tmp/cifar_{name}.beton',
     29                             batch_size=BATCH_SIZE,
     30                             num_workers=8,

3 frames
[/usr/local/lib/python3.10/dist-packages/ffcv/loader/loader.py](https://localhost:8080/#) in __init__(self, fname, batch_size, num_workers, os_cache, order, distributed, seed, indices, pipelines, custom_fields, drop_last, batches_ahead, recompile)
    142             self.num_workers = cpu_count()
    143 
--> 144         Compiler.set_num_threads(self.num_workers)
    145 
    146         if indices is None:

[/usr/local/lib/python3.10/dist-packages/ffcv/pipeline/compiler.py](https://localhost:8080/#) in set_num_threads(cls, n)
     18             n = cpu_count()
     19         cls.num_threads = n
---> 20         set_num_threads(n)
     21         ch.set_num_threads(n)
     22 

[/usr/local/lib/python3.10/dist-packages/numba/np/ufunc/parallel.py](https://localhost:8080/#) in set_num_threads(n)
    606     if not isinstance(n, (int, np.integer)):
    607         raise TypeError("The number of threads specified must be an integer")
--> 608     snt_check(n)
    609     _set_num_threads(n)
    610 

[/usr/local/lib/python3.10/dist-packages/numba/np/ufunc/parallel.py](https://localhost:8080/#) in snt_check(n)
    568     def snt_check(n):
    569         if n > NUMBA_NUM_THREADS or n < 1:
--> 570             raise ValueError(msg)
    571     return snt_check
    572 

ValueError: The number of threads must be between 1 and 2```
levongh commented 4 months ago

@AntonioMacaronio try to change the ffcv version, from the backtrace it seems the ffcv version is pretty old. the code is changed by the following commit https://github.com/libffcv/ffcv/commit/352d1ef183aa681f76a19420dcd764fa680b1d1e