More detailed data api instruction ...

htlee6 commented 3 years ago

Hi there,

Now I have my directory BNN-ch21 like this

.
|-- README.md
|-- dataset
|   |-- gnnet-ch21-dataset-train
|   |   |-- 25
|   |   |-- 30
|   |   |-- 35
|   |   |-- 40
|   |   |-- 45
|   |   `-- 50
|   |-- gnnet-ch21-dataset-validation
|   |   |-- ch21-val-setting-1
|   |   |-- ch21-val-setting-2
|   |   `-- ch21-val-setting-3
|   |-- train.tar.gz
|   `-- validation.tar.gz
|-- dataset_prep.sh
|-- docker
|   |-- Dockerfile
|   |-- Dockerfile-cuda-11.1
|   |-- docker_start.sh
|   |-- environment.yml
|   `-- ssh_config
|-- model
|-- test.py
`-- util
    |-- __pycache__
    |   |-- datanetAPI.cpython-35.pyc
    |   `-- datanetAPI.cpython-38.pyc
    `-- datanetAPI.py

And I wrote in my test.py like this to test the data pipeline api,

from util import datanetAPI as api

reader = api.DatanetAPI(data_folder="/BNN-ch21/dataset/gnnet-ch21-dataset-train",
                        intensity_values=[25],
                        shuffle=False)

it = iter(reader)
for sample in it:
    print(sample)

And the output is always a stack of error loggings like

Error in the file: results_30_400-2000_9925_9949.tar.gz   iteration: 0
Error in the file: results_30_400-2000_9950_9974.tar.gz   iteration: 0
Error in the file: results_30_400-2000_9975_9999.tar.gz   iteration: 0
...

Could you help me by offering more detailed instructions?

Thanks in advance!

albert-lopez commented 3 years ago

Hello,

Is it possible that you are using the wrong API version? For the GNNet challenge 2021 you should use: https://github.com/BNN-UPC/datanetAPI/tree/challenge2021. On the other hand, to filter topology sizes you should use the parameter TopologySizes . Intensity_values is used to filter the traffic intensity range that in this dataset goes from 400 to 2000.

Regards

htlee6 commented 3 years ago

Aha! Great point! Solved my question, thanks!

BNN-UPC / datanetAPI

More detailed data api instruction ... #3