mitchellnw
commented
5 years ago No problem,
What you could do:
is in charge of reading the HDF5 file.
When it is initialized, it is given a link to this HDF5 file. It is initialized here:
where images_file_name initially comes from args.
So if you want to use different features you should add
...
--images_file_name <new-features>
...to your run.
You should place these features in thor_offline_data/FloorPlan<n>/<new-features>.
The images are read here
so there should be a feature associates with each str(self.state).
The str method is here:
What do we do:
We run the following script:
import json
import os
import time
import warnings
from collections import deque
from math import gcd
from multiprocessing import Process, Queue
from ai2thor.controller import BFSController
from datasets.offline_controller_with_small_rotation import ExhaustiveBFSController
def search_and_save(in_queue):
while not in_queue.empty():
try:
scene_name = in_queue.get(timeout=3)
except:
return
c = None
try:
out_dir = os.path.join(<path-to-where-you-want-data-to-go>, scene_name)
if not os.path.exists(out_dir):
os.mkdir(out_dir)
print('starting:', scene_name)
c = ExhaustiveBFSController(
grid_size=0.25,
fov=90.0,
grid_file=os.path.join(out_dir, 'grid.json'),
graph_file=os.path.join(out_dir, 'graph.json'),
metadata_file=os.path.join(out_dir, 'metadata.json'),
images_file=os.path.join(out_dir, 'images.hdf5'),
depth_file=os.path.join(out_dir, 'depth.hdf5'),
grid_assumption=False)
c.start()
c.search_all_closed(scene_name)
c.stop()
except AssertionError as e:
print('Error is', e)
print('Error in scene {}'.format(scene_name))
if c is not None:
c.stop()
continue
def main():
num_processes = 30
queue = Queue()
scene_names = []
for i in range(2):
for j in range(30):
if i == 0:
scene_names.append("FloorPlan" + str(j + 1))
else:
scene_names.append("FloorPlan" + str(i + 1) + '%02d' % (j + 1))
for x in scene_names:
queue.put(x)
processes = []
for i in range(num_processes):
p = Process(target=search_and_save, args=(queue,))
p.start()
processes.append(p)
for p in processes:
p.join()
Note that AI2-THOR (https://github.com/allenai/ai2thor) has changed since we began this project so this script will not work out of the box -- it may require some changes. I am sorry about this and am happy to help if there are issues. The scenes in Thor themselves have also changed and are much better now.
This script does BFS in a scene and saves an HDF5 file in the desired format with all of the RGB images.
e.g. you will now have FloorPlan<n>/images.hdf5.
You can now simply iterate over these files to get the features that you want. We run a variant of the following script to get the ResNet features:
for scene in scenes:
images = h5py.File('{}/{}/images.hdf5'.format(data_dir, scene), 'r')
features = h5py.File('{}/{}/{}.hdf5'.format(data_dir, scene, method), 'w')
for k in images:
frame = resnet_input_transform(images[k][:], 224)
frame = torch.Tensor(frame)
if torch.cuda.is_available():
frame = frame.cuda()
frame = frame.unsqueeze(0)
v = model(frame)
v = v.view(512, 7, 7)
v = v.cpu().numpy()
features.create_dataset(k, data=v)
images.close()
features.close()where resnet_input_transform is
def resnet_input_transform(input_image, im_size):
normalize = transforms.Normalize(
mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
all_transforms = transforms.Compose([
transforms.ToPILImage(),
ScaleBothSides(im_size),
transforms.ToTensor(),
normalize,
])
transformed_image = all_transforms(input_image)
return transformed_image
AruniRC
xuai05
Hi,
can you please share the file format and structure for storing image features?
It seems your code reads in pre-computed features from an HDF5 dump for each frame that the agent can see in this line in your codebase: https://github.com/allenai/savn/blob/1cda8aff1722543450e72bc74374da80dc84c771/datasets/environment.py#L18
If we want to use other types of features (e.g. a different feature extractor than what you use to represent the frame image), it would be super helpful to have more details on how the feature dump is constructed, the ordering of the frame images, etc.
thank yoU!