Information on how to run the experiments

[yannickl96]

Hello there,

would it be possible to provide additional information on how to run your experiments? Up until now I managed to obtain the godrive executable, however, when I try to use the run.sh from the repository I get an error that train_5.npy is not found. I went ahead and downloaded the dataset from https://github.com/felipessalvatore/self_driving_data, but there is only train_data.npy.

Any advice on running training and test classification would be appreciated.

Best regards!

[RenatoGeh]

Hi Yannick,

The data were generated on top of Salvatore's original dataset (which you pointed out) following these scripts: [1].

I admit there is next to no documentation, so apologies for that. I'll try to cover the basics so you're able to reproduce my experiments. :)

We have the following scripts in [1]:

1. transform.py takes the original dataset, converts images to grayscale, optionally reduces resolution, and does some basic data augmentation (mirror the vertical axis to get more left and right images).
2. all_transform.sh runs transform.py on the three datasets: train, test and valid.
3. quantize.py takes a dataset and lowers the resolution, scaling all images to some max value.
4. equalize.py attempts to "flatten" a dataset's histogram of possible values. This is because LearnSPN learns the structure of a probabilistic circuit by running several instances of an independence test on sub-datasets, and contingency table indep tests are less reliable when there are few instances of a particular value.
5. binarize.py binarizes a dataset via Otsu binarization.
6. view_images.py is just a utility script for plotting images of a dataset.
7. save_samples.py saves randomly drawn samples from a dataset as .png image files.
8. save_merged_samples.py does the same as save_samples.py, but merges images into a single file.
9. print_histogram.py plots a dataset's histogram (of possible grayscale values).

So ideally you'd want to download the original dataset from Salvatore's repo, apply transformations using the scripts above, and finally run main.go with the desired options, passing your datasets as input.

To run main.go, you may either compile and run the binary, or run directly with go run. Running with no arguments gives a list of options. In short, GoDrive either

1. Tests: this is the real deal. It assumes you're running from within a Raspberry Pi connected (through USB) to a NXT Lego Mindstorm Brick. You should probably have the Brick ready and running the Java part [5] which controls the motors.
2. Trains: runs the given structure and parameter learning algorithms and outputs a model as a binary file.
3. Samples: this is basically for data collecting. You may want to do this first and generate your own dataset. One problem we found was that, if your bot has a different camera position (with relation to the ground/bot) then the training dataset's, your classifier might struggle to get anything right. I said it wrong here. Sample is actually for measuring accuracy and response time of a model on a dataset. The part where I talk about camera position still stands though.

run.sh was a utility script I wrote to help with benchmarking and testing. If you want to replicate the experiments, you'll need to change the arch and learn arrays to cover all structure and parameter learners we covered in our article and thesis [2][3]. Once run.sh has finished running, you'll get several .mdl, which are binary serializations of each probabilistic circuit learned from their respective algorithms (you can find more about this here [4]). These .mdl's should be loaded from within the Raspberry Pi, using GoDrive's main.go.

Hope that helps, Renato

PS: I'll leave this issue open as some kind of documentation.

[1] - https://github.com/RenatoGeh/datasets/tree/master/self_driving [2] - https://sol.sbc.org.br/index.php/eniac/article/view/9292 [3] - https://www.ime.usp.br/~renatolg/mac0499/docs/thesis.pdf [4] - https://github.com/RenatoGeh/godrive/tree/master/models [5] - https://github.com/RenatoGeh/godrive/blob/master/java/Remote.java

[yannickl96]

Hello Renato,

thanks for your very elaborate reply! Just one further question for now: You mention that the tests option in GoDrive runs on the Pi with the NXT Lego Mindstorm Brick. If I understood correctly, in your paper you did not run all different preprocessings + learning combinations on the robot, but just the ones shown in table 3. I personally would be more interested to reproduce your results from table 2 which were tested on your desktop computer IIRC. So, if I change the arch and learn arrays, I can obtain the models from table 2 as .mdl files. Which script did you use to run the tests on these models in order to obtain the accuracies in table 2?

Thanks in advance!

Best, Yannick

[RenatoGeh]

If I understood correctly, in your paper you did not run all different preprocessings + learning combinations on the robot, but just the ones shown in table 3.

You're right. Some preprocessings were too slow to run on the Pi to have a quick enough response in the Brick.

I personally would be more interested to reproduce your results from table 2 which were tested on your desktop computer IIRC. So, if I change the arch and learn arrays, I can obtain the models from table 2 as .mdl files. Which script did you use to run the tests on these models in order to obtain the accuracies in table 2?

I just went digging into the code and realized I said something wrong. The sample options for main.go is not for generating a dataset, but to generate exactly what you're looking for. It takes a dataset and a .mdl file and outputs accuracy and time it took to evaluate all of the dataset.

With regard to the data, honestly, I can't find the scripts to generate all datasets with all image transformations. You'll probably have to apply transformations manually and then run sample.

[yannickl96]

Thanks for the clarification! I will give it a shot :)

[yannickl96]

Ok maybe you can give me a pointer where I might be doing something wrong: I am trying to reproduce the Q4 line, GD+d column of Table 2 in the paper. I did the following:

1. Downloaded the scripts from https://github.com/RenatoGeh/datasets/tree/master/self_driving.
2. Copied the original dataset to the folder containing the python scripts.
3. Created a new directory transformed and used the all_transform.sh script to apply the grayscaling to all datasets.
4. Then I used python quantize.py transformed/train.npy 256 16 transformed/q4_train.npy to apply 4-bit quantization to the training set.
5. Same as above, but for testing set.
6. Copied those q4_*.npy files over to GoDrive where I created a folder datasets.
7. Call go run main.go r q4_gd_d.mdl gens d 500 15 datasets/q4_train.npy to train the model according to what was said in the paper.
8. Call go run main.go s q4_gd_d.mdl gens 15 datasets/q4_test.npy to test the model on the testset. Compared to the paper (which states accuracy of 78.2%) I only get Accuracy: 72.4% in the output of the last command. Did I overlook some setting or use some wrong argument in the call to main.go or the preprocessing? I was quite confused by the max pixel values as you subtract 1 in the quantize.py script but add 1 in main.go.

Best, Yannick

[RenatoGeh]

It has been some time since I coded this project, so I'm not sure if you have to subtract or add wherever. One thing that does impact accuracy is an unbalanced dataset. We balanced the dataset by having all labels have equal number of images. You might want to check your dataset and balance if it's the case.

[yannickl96]

Hi Renato,

thank you again for your reply! I went through the code and found that the balancing of the datasets (for the trainset, as well as for the test set) should be correct, since the dataset is loaded using readBalanced from GoSPN. When you run main.go for training or Sample, are the samples drawn deterministically from the numpy dataset? So for example for each call to go run main.go s q4_gd_d.mdl gens 15 datasets/q4_test.npy, are the 500 drawn samples the same?

Best, Yannick

[RenatoGeh]

Yes, they are deterministically drawn. Are the dataset sizes for both training and testing the same as we used?

In any case, I went digging through my original code and found the data we used as well as the trained models. I'm not sure if everything is in there, but might as well share. https://drive.google.com/drive/folders/1OP47IETaqGcUdLoXUkGWkWEQzvhz91K-?usp=sharing

[yannickl96]

Hi Renato,

thanks for sharing! I just went through the files. As far as I can tell, backup/gens_d_1_500.txt should correspond to the results of running go run main.go s backup/saved/gens_d_1_500.mdl gens 1 data/test_1.npy and the output should correspond to the B-row column GD+d in table 2 of the paper, as the .txt-file shows an accuracy of 83.8%. I checked that data/data.go and data/data_test.go are identical to the ones in this repository (which they are). But I unfortunately only get Accuracy: 0.73200. I also tried the gens_disc_bin.mdl in the backup/saved-folder which gave an accuracy of 33.33%. Am I using the wrong testset or wrong .mdl file? I also checked that the number of examples for each label in the 500 samples for testing is equal (167,167,166).

Best, Yannick

[RenatoGeh]

That's weird. It might have been a bug introduced into the GoSPN library (where the actual learning and inference is done) after we did the experiments in GoDrive. As you can see here (https://github.com/RenatoGeh/gospn/commits/dev) I pushed a bunch of commits after we did the experiments. One of them might have broken something.

[yannickl96]

Hi Renato, thanks for pointing that out. Could you tell me what was the last commit before running the experiments? Then I would try that one. Since the paper was published in October 2019 and the latest commit in gospn/dev is from January 2019 it is hard for me to see that myself. I tried commit d9047398916321e298329561ff2b9712d78bde3d in gospn/dev but that still yields the same results as my previous post.

Best, Yannick

[RenatoGeh]

The paper's from mid to late 2019, but the data is from late 2018 as it was originally my undergraduate final project. You might want to try to git checkout GoSPN around that time.

You might want to use git bisect to automatically find the bad commit.