Extracting features for high performing model that is already trained

[ardila]

@daseibert

When I tried to write features:

python shownet.py -f /tmp/ConvNet__2013-11-22_15.25.16/52.3948 --write-features=rnorm5_15a, rnorm5_15b --feature-path=/batch_storage/diego_features

I got the following error:

here
pdir: /tmp
self.save_path:  /tmp
Traceback (most recent call last):
  File "shownet.py", line 505, in <module>
    model = ShowConvNet(op, load_dic)
  File "shownet.py", line 48, in __init__
    ConvNet.__init__(self, op, load_dic)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/convnet.py", line 46, in __init__
    IGPUModel.__init__(self, "ConvNet", op, load_dic, filename_options, dp_params=dp_params)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/gpumodel.py", line 86, in __init__
    self.init_data_providers()
  File "shownet.py", line 66, in init_data_providers
    ConvNet.init_data_providers(self)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/gpumodel.py", line 117, in init_data_providers
    type=self.dp_type, dp_params=self.dp_params, test=True)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 121, in get_instance
    return _class(data_dir, batch_range, init_epoch, init_batchnum, dp_params, test)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/convdata.py", line 245, in __init__
    LabeledMemoryDataProvider.__init__(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 215, in __init__
    LabeledDataProvider.__init__(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 208, in __init__
    DataProvider.__init__(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 36, in __init__
    batch_range = DataProvider.get_batch_nums(data_dir)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 146, in get_batch_nums
    names = DataProvider.get_batch_filenames(srcdir)
  File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 142, in get_batch_filenames
    return sorted([f for f in os.listdir(srcdir) if DataProvider.BATCH_REGEX.match(f)], key=alphanum_key)
OSError: [Errno 2] No such file or directory: '/batch_storage/batch128_img138_full/'

I believe this is either because I have the wrong model file, or because you changed the name of the folder of the batch storage. Let me know what would be a safe way of fixing this.

[daseibert]

I moved /batch_storage/batch128_img138_full to /batch_storage2/batch128_img138_full because I'm writing out the full imagenet to that drive and wanted all space available on that disk.

On Sun, Dec 1, 2013 at 12:50 PM, Diego Ardila notifications@github.comwrote:

@daseibert https://github.com/daseibert

When I tried to write features:

python shownet.py -f /tmp/ConvNet__2013-11-22_15.25.16/52.3948 --write-features=rnorm5_15a, rnorm5_15b --feature-path=/batch_storage/diego_features

I got the following error:

here pdir: /tmp self.save_path: /tmp Traceback (most recent call last): File "shownet.py", line 505, in model = ShowConvNet(op, load_dic) File "shownet.py", line 48, in init ConvNet.init(self, op, load_dic) File "/home/ardila/src/archconvnets/archconvnets/convnet/convnet.py", line 46, in init IGPUModel.init(self, "ConvNet", op, load_dic, filename_options, dp_params=dp_params) File "/home/ardila/src/archconvnets/archconvnets/convnet/gpumodel.py", line 86, in init self.init_data_providers() File "shownet.py", line 66, in init_data_providers ConvNet.init_data_providers(self) File "/home/ardila/src/archconvnets/archconvnets/convnet/gpumodel.py", line 117, in init_data_providers type=self.dp_type, dp_params=self.dp_params, test=True) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 121, in get_instance return _class(data_dir, batch_range, init_epoch, init_batchnum, dp_params, test) File "/home/ardila/src/archconvnets/archconvnets/convnet/convdata.py", line 245, in init LabeledMemoryDataProvider.init(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 215, in init LabeledDataProvider.init(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 208, in init DataProvider.init(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 36, in init batch_range = DataProvider.get_batch_nums(data_dir) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 146, in get_batch_nums names = DataProvider.get_batch_filenames(srcdir) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 142, in get_batch_filenames return sorted([f for f in os.listdir(srcdir) if DataProvider.BATCH_REGEX.match(f)], key=alphanum_key) OSError: [Errno 2] No such file or directory: '/batch_storage/batch128_img138_full/'

I believe this is either because I have the wrong model file, or because you changed the name of the folder of the batch storage. Let me know what would be a safe way of fixing this.

— Reply to this email directly or view it on GitHubhttps://github.com/dicarlolab/archconvnets/issues/8 .

[daseibert]

also, fyi, shownet.py uses the gpu too, but does not check if the gpu is already being used. It will occasionally crash pre-existing jobs if its default gpu is already being used. It's safer to manually specify the gpu number.

On Sun, Dec 1, 2013 at 1:09 PM, Darren Seibert darren@mit.edu wrote:

I moved /batch_storage/batch128_img138_full to /batch_storage2/batch128_img138_full because I'm writing out the full imagenet to that drive and wanted all space available on that disk.

On Sun, Dec 1, 2013 at 12:50 PM, Diego Ardila notifications@github.comwrote:

@daseibert https://github.com/daseibert

When I tried to write features:

python shownet.py -f /tmp/ConvNet__2013-11-22_15.25.16/52.3948 --write-features=rnorm5_15a, rnorm5_15b --feature-path=/batch_storage/diego_features

I got the following error:

here pdir: /tmp self.save_path: /tmp Traceback (most recent call last): File "shownet.py", line 505, in model = ShowConvNet(op, load_dic) File "shownet.py", line 48, in init ConvNet.init(self, op, load_dic) File "/home/ardila/src/archconvnets/archconvnets/convnet/convnet.py", line 46, in init IGPUModel.init(self, "ConvNet", op, load_dic, filename_options, dp_params=dp_params) File "/home/ardila/src/archconvnets/archconvnets/convnet/gpumodel.py", line 86, in init self.init_data_providers() File "shownet.py", line 66, in init_data_providers ConvNet.init_data_providers(self) File "/home/ardila/src/archconvnets/archconvnets/convnet/gpumodel.py", line 117, in init_data_providers type=self.dp_type, dp_params=self.dp_params, test=True) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 121, in get_instance return _class(data_dir, batch_range, init_epoch, init_batchnum, dp_params, test) File "/home/ardila/src/archconvnets/archconvnets/convnet/convdata.py", line 245, in init LabeledMemoryDataProvider.init(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 215, in init LabeledDataProvider.init(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 208, in init DataProvider.init(self, data_dir, batch_range, init_epoch, init_batchnum, dp_params, test) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 36, in init batch_range = DataProvider.get_batch_nums(data_dir) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 146, in get_batch_nums names = DataProvider.get_batch_filenames(srcdir) File "/home/ardila/src/archconvnets/archconvnets/convnet/data.py", line 142, in get_batch_filenames return sorted([f for f in os.listdir(srcdir) if DataProvider.BATCH_REGEX.match(f)], key=alphanum_key) OSError: [Errno 2] No such file or directory: '/batch_storage/batch128_img138_full/'

I believe this is either because I have the wrong model file, or because you changed the name of the folder of the batch storage. Let me know what would be a safe way of fixing this.

— Reply to this email directly or view it on GitHubhttps://github.com/dicarlolab/archconvnets/issues/8 .

[ardila]

This was solved by loading data from the right place.