search

google-research / snap

SNAP: Self-supervised Neural Maps for Visual Positioning and Semantic Understanding (NeurIPS 2023)
Apache License 2.0
180 stars 16 forks source link

RastersConfig in types.py missed #7

Open ZhouMengjie opened 2 months ago

ZhouMengjie commented 2 months ago

Hi, I found that in snap/data/types.py, line 133, the class RastersConfig() is not defined in the code. Could you please provide these information for rasters configuration? Additionaly, could I know whether these setting are relevant to "aerial" or "semantic" modality as map input? Many thanks for the help.

etrulls commented 2 months ago

This is a config dict for the overhead rasters we used in the paper. IIRC there were two types: RGB rasters (the "aerial" modality) used as model inputs along with the ground-level images, and semantic rasters ("semantic" modality) used only to train the "orienternet" baseline in Table 1.

I think you should be able to disable this, unless you really have aerial rasters aligned with the BEV grid. If you do, I think the only relevant parameter in this config is "pixel_size", which is the size of a pixel in the aerial raster, in meters (that was 20cm in our case, as stated in the paper).

ZhouMengjie commented 2 months ago

Thanks so much for your reply, Eduard. I also noted that in loader.py, line 431 and 432. The "semantic_map_classes" and "semantic_classes_gt" also need to be set in rasters_config. Could you please tell me more about these information? Are they useful for subsequent processing in ground-level and aerial branch?

etrulls commented 2 months ago

Those are just tuples with a list of strings with the semantic classes (e.g. "tree", "road", etc).

The first one ("semantic_map_classes") corresponds to the input used for the orienternet baseline (i.e. not used by our model). The second one ("gt_semantic_classes") corresponds to the training data and ground truth for the "semantic mapping" experiments using our features as pretraining (i.e. Fig. 6) -- you don't need them to run the localization training/evals. Both are specific to our data.

ZhouMengjie commented 2 months ago

Hi, thank you very much for your prompt support. I’m quite interested in your work and am actively working to train and test it on our own dataset. However, I am a bit confused about the dataset structure and what should I input.

I noticed that loading the data involves using a builder with the "builder = tfds.core.read_only_builder.builder_from_directories(dirs)". It seems that you’ve already registered your dataset with TensorFlow, allowing it to automatically read the dataset_info.json from the specific directories.

Could you clarify whether you created a custom dataset as described here: https://www.tensorflow.org/datasets/add_dataset? If so, could you provide the code (e.g., info, split_generators, __generate_examples) you used to define this custom dataset?

Please feel free to omit any sensitive information. I’m just particularly interested in understanding what information needs to be included in metadata and feature dict in info() and how to properly format the examples in generate_examples() to ensure they work correctly with the model.

I greatly appreciate any help you can provide.

etrulls commented 2 months ago

We didn't publish that part of the code because it would be pretty difficult to disentangle from our internal infra, and because it wouldn't be useful anyway to train on different data. You can freely remove all of this and use whatever format you prefer to feed examples to the training / eval loops. I'll send you an email with a couple more details.