search

puppet-meteor / MINER

MINER provided by the paper "MINER: A Hybrid Data-Driven Approach for REST API Fuzzing"
36 stars 6 forks source link

MINER attentionmodel_group questions #8

Open laxman2405 opened 7 months ago

laxman2405 commented 7 months ago

Hi,

I am pretty much new to RESTler and wanted to explore on it. So, we have our own REST API and we gave it to RESTler and it was able to give out some vulernabilities in the APIs. I tried to replicate the same with MINER as well. When I ran the Fuzz command with MINER, this also gives same number of vulnerabilities.

So, my question in the attentionmodel_group is, there are 2 files attentio.py and generation.py. In these two files, there are multiple .npy, .pkl files used and loaded using numpy. Couple of questions on this.

  1. Do these files generate automatically under the folder?
  2. How do we create these files according to our API specification. Any docs that I can refer to will be helpful as it was not mentioned how to use those in the READme file.

Appreciate your help in advance!

laxman2405 commented 7 months ago

Any update on how to do this, please?

m0ck1ng commented 6 months ago

Hi Laxman, sorry for the late reply.

The training module in MINER, like attention.py and generation.py, do require .pkl, .npy files. We would like to clarify that these files are automatically collected during fuzzing campaign by a collection module without any pre-positioned datasets. When fuzzing REST APIs, the test cases regarded as interesting (e.g. passing server's check, triggering errors) will be preserved and compose a dataset (c.f. traingset.pkl). The further operations then will extract .pkl, .npy from the dataset.

We suggest you read the Chapter3.2 in MINER's paper carefully, where the process of training and collection is detailed.

In addition, the file paths in MINER's codebase are the absolute paths and hardcoded in the source codes. I have no idea whether it causes any trouble.

laxman2405 commented 5 months ago

I did run the MINER in our windows machine by changing the path from /home/MINER to our local path from windows. But still, the .pkl and .npy files are not generated. Any file that we can check to debug or find the reason behind this?

Things we did:

  1. We had our own swagger json file and we generated the Compile folder from RESTler and placed the Compile folder in MINER and ran the Restler test command, the files didn't get generated. Is there any other step that we are missing here?
laxman2405 commented 5 months ago

Any inputs on the query? Does MINER run in windows ?

Lawrenmeta commented 5 months ago

Any inputs on the query? Does MINER run in windows ?

Lately I also try to explore on restler , i want to konw how could you run miner in windows , which part will be changed?

laxman2405 commented 5 months ago

I have just changed the paths that are hard coded in the attention model python files and thought it would run on windows. The fuzz command is running, but my expectation is, after we run the fuzz command, it should generate bunch of .pkl, .npy and other files that are mentioned in those python files in the given path. But, even when run fuzz for 48hrs, it won't generate any files under that folder. So, my assumption is MINER will not run on windows. Correct me if my assumption is wrong.

puppet-meteor commented 5 months ago

I have just changed the paths that are hard coded in the attention model python files and thought it would run on windows. The fuzz command is running, but my expectation is, after we run the fuzz command, it should generate bunch of .pkl, .npy and other files that are mentioned in those python files in the given path. But, even when run fuzz for 48hrs, it won't generate any files under that folder. So, my assumption is MINER will not run on windows. Correct me if my assumption is wrong.

Hi, it's been more than a year since we last use MINER, so the details are already been forgot. I've forgetten whether the compile process works on windows, while the compile process must work on Ubuntu. Could you provide MINER runtime log files for your case? BTW, why you run MINER on windows? As we fuzz a target cloud service through APIs, the tested cloud service is insensitive to the OS of the testing machine. You guys can run MINER on Ubuntu if windows machince is not necessary.

laxman2405 commented 5 months ago

I have just changed the paths that are hard coded in the attention model python files and thought it would run on windows. The fuzz command is running, but my expectation is, after we run the fuzz command, it should generate bunch of .pkl, .npy and other files that are mentioned in those python files in the given path. But, even when run fuzz for 48hrs, it won't generate any files under that folder. So, my assumption is MINER will not run on windows. Correct me if my assumption is wrong.

Hi, it's been more than a year since we last use MINER, so the details are already been forgot. I've forgetten whether the compile process works on windows, while the compile process must work on Ubuntu. Could you provide MINER runtime log files for your case? BTW, why you run MINER on windows? As we fuzz a target cloud service through APIs, the tested cloud service is insensitive to the OS of the testing machine. You guys can run MINER on Ubuntu if windows machince is not necessary.

Yes, we did run miner on ubuntu 20.04 as well. But we modified the Compile folder and added our own Compile folder that got generated after compiling with our own swagger json file. We had our own application which was running the background in local in ubuntu and the we triggered restler fuzz command. Is it required that the application has to run on cloud and not in local?

Also, Miner is using gitlab apis in the compile folder. Can we run miner directly after we clone without any changes? Does it hit the gitlab apis and identify the bugs?

puppet-meteor commented 5 months ago

running apps on cloud is a more reasonable and practical way. For your second question, I've forgetten the details. It should be the ready-to-use version. Again, you can provide the runtime log file to see if error occurs during MINER running.

laxman2405 commented 5 months ago

We did run the miner code as is after cloning and we get these in the logs. network.testing.140070828668672.1.txt

puppet-meteor commented 5 months ago

DataDrivenChecker Works as shown in logs, so MINER's design works. While this log is too short to see the training process. BTW, the execution speed is pretty slow. It takes 2 mins for a sequence testing, which is different compared to our experiments.

laxman2405 commented 5 months ago

Also, how does Miner uses attention.py and generation.py files. Do they get called automatically? Generation.py has a line to load word_to_ix.npy file as first step. Will that file be generated as part of fuzzing process?

Just by running restler fuzz command, does those python files gets called automatically or we have to manually run them ?

please help me with detailed steps as how to run miner, is it just running fuzz command? Then how does those puthon files gets called and when?

laxman2405 commented 5 months ago

Can you please help me with those questions?

laxman2405 commented 5 months ago

Is there any update on this issue?

puppet-meteor commented 5 months ago

Also, how does Miner uses attention.py and generation.py files. Do they get called automatically? Generation.py has a line to load word_to_ix.npy file as first step. Will that file be generated as part of fuzzing process?

Just by running restler fuzz command, does those python files gets called automatically or we have to manually run them ?

please help me with detailed steps as how to run miner, is it just running fuzz command? Then how does those puthon files gets called and when?

yep, attention.py and generation.py are called automatically. The related code is in MINER/restler_bin_atten/engine/engine/core/datacollect.py. Other codes like 'MINER/restler_bin_atten/engine/engine/core/requests.py' and 'MINER/restler_bin_atten/engine/engine/core/sequences.py' will automatically call the functions in datacollect.py during fuzzing. You can read MINER‘s paper and read codes to figure out the call logic.

laxman2405 commented 5 months ago

Thank you for the update. We can still replace Compile folder with our own, that got generated with our own json file, right?

puppet-meteor commented 5 months ago

If you don't change the related codes in datacollect.py/requests.py/sequences.py, and the compile folder you replaced does not influence the execution logic of base RESTler, then I think the functionality of MINER works normally.

However, if the functionality is normal, then additional files, such as dictrequestid.pkl and word_to_ix.npy, should be generated.

laxman2405 commented 5 months ago

okay, is there any minimum time_buget we have to specify. We know that, in READme it's mentioned 12 hrs and in the research paper i guess it was run for 48hrs. So, what can we have as minimum time budget to generate those files?