search

blue-oil / blueoil

Bring Deep Learning to small devices
https://blueoil.org
Apache License 2.0
248 stars 86 forks source link

Support GCS I/O #1205

Open hadusam opened 4 years ago

hadusam commented 4 years ago

In Blueoil, we can train with GCS input and save training results to GCS, but other operations (e.g. convert, evaluate...) are not supported with GCS input/output.

Issues

There are many files and operations that depend on input and output directly.

blueoil/cmd/*.py
blueoil/utils/config.py
blueoil/utiles/image.py
blueoil/utils/predict_output/writer.py
blueoil/converter/template.py
blueoil/converter/frontend/tensorflow.py
blueoil/converter/code_generation.py

and so on...

We can replace them to support GCS with tensorflow.io.gfile, but some operations need to rewrite several lines. So it is better to add utils of operations of I/O and manage them in a single module.

Targets

There are operations with files/directories directly.

os.path.exists
os.path.abspath
os.makedirs
shutil.copy
shutil.rmtree
open()
np.save()
PIL.Image.save()
PIL.Image.open()

Since it is difficult to manage all of these operations in individual files and manage them separately, it is better to provide a module that aggregates the operations that perform file I/O and change it to use this module for all except special cases.

hadusam commented 4 years ago

Tasks

Support GCS I/O in each operation