[RE] Effective Program Debloating via Reinforcement Learning

[Kahss]

Original article: Effective Program Debloating via Reinforcement Learning

PDF URL: https://gitlab.com/onepoint/recherche-et-developpement/optimus/chisel-replication/-/blob/master/Replication_CHISEL.pdf Metadata URL: https://gitlab.com/onepoint/recherche-et-developpement/optimus/chisel-replication/-/blob/master/metadata.yaml Code URL: https://gitlab.com/onepoint/recherche-et-developpement/optimus/chisel-replication/-/tree/master

Scientific domain: Program Debloating and Machine Learning Programming language: Python Suggested editor: Visual Studio Code or VIM

[Kahss]

Thank you very much Esther, your comments are really insightful. I will work on the paper in the few next days and come back with an updated version. Thanks again !

[Kahss]

Hello again @ESSYCHAN , Thanks again for your comments. I've been integrating them as precisely as possible in the paper sources. The gitlab repository has been adequately updated.

I would have some questions about your remarks on the code:

• When you say that the main.py file produces an error, could you please give me the command you're trying to run ? The file is not meant to be run directly, but rather through the indirect command $make run_example TARGET='' which itself calls $python -m optimus -c examples//config.yaml
• I'm not sure to understand your remarks regarding the test cases resetting to range (3). Could you please detail it ?
• About the command $make example, it's possibly a typo, the right command being $make examples. Could you please confirm that it's just a typo problem ?
• Thank you very much for your kind return. Lots of efforts have been put in the code organization.

[Kahss]

Dear @wmkouw , your comments have also been integrated, thanks again.

[wmkouw]

@ESSYCHAN @wmkouw are your reviews possible to be done by the end of the month?

I'm sorry. I don't have the time to review the paper again. I recommended to accept the paper the first time around and I trust @ESSYCHAN's feedback to have led to improvements, so I'd recommend 'accept' again.

[ESSYCHAN]

Hello again @ESSYCHAN , Thanks again for your comments. I've been integrating them as precisely as possible in the paper sources. The gitlab repository has been adequately updated.

I would have some questions about your remarks on the code:

• When you say that the main.py file produces an error, could you please give me the command you're trying to run ? The file is not meant to be run directly, but rather through the indirect command $make run_example TARGET='' which itself calls $python -m optimus -c examples//config.yaml
• I'm not sure to understand your remarks regarding the test cases resetting to range (3). Could you please detail it ?
• About the command $make example, it's possibly a typo, the right command being $make examples. Could you please confirm that it's just a typo problem ?
• Thank you very much for your kind return. Lots of efforts have been put in the code organization.

Hello,

For the main file, I tried just running the file on its own, but it worked well when I ran it indirectly. $make run_example TARGET yes this was a typo problem, it's running well with $make examples As for the range(3), I tried to increase to range 10 but as soon as the file runs, the range resets to the original value 3.

I believe that given the corrections you've made I would recommend a pass again.

[Kahss]

@wmkouw Thank you very much for your approval.

@ESSYCHAN Ok I understand. Actually to be sure that examples can be run whatever the potential previous execution errors, I use a <filename>_CLEAN.py version for the ressource files (in the ressource folder). This way, to ensure that everything is ok, I erase the last copy of the file and I duplicate the CLEAN version. I hope it helps. And anyway, thank you very much for your approval.

[Kahss]

@oliviaguest , could you please tell me what is the next step of the publication process ? Do you require a third reviewer for the paper to be accepted ?

[rougier]

@D-Maurel We need only two reviewer such that you paper is virtually accepted but @oliviaguest will confirm (or not)

[Kahss]

Thanks for the precisions @rougier. I can't wait to see the acceptance confirmed !

[rougier]

@oliviaguest Can we proceed with publication?

[oliviaguest]

@rougier I have been and still am very unwell for the past 3 weeks. I can try and get to this soon, but unlikely to be today as I am playing catch up. Apologies.

[rougier]

@oliviaguest No problem. As soon as we got your go, I can publish the article.

[oliviaguest]

@wmkouw Thank you very much for your approval.

@ESSYCHAN Ok I understand. Actually to be sure that examples can be run whatever the potential previous execution errors, I use a <filename>_CLEAN.py version for the ressource files (in the ressource folder). This way, to ensure that everything is ok, I erase the last copy of the file and I duplicate the CLEAN version. I hope it helps. And anyway, thank you very much for your approval.

Is this clear now in the documentation, @D-Maurel?

[Kahss]

@oliviaguest , yes it is now. I should have modified the README earlier, my bad.

[oliviaguest]

@D-Maurel @ESSYCHAN is there anything left unaddressed from Esther's feedback here? If not, @ESSYCHAN could you check things that have been made sense for you? Your feedback has been super valuable and I'd hate for it not be used to its full extent. ☺️

[Kahss]

@oliviaguest , for me everything has been addressed. @ESSYCHAN please tell me if there still is any unclear spot in the paper or in the code.

[rougier]

@oliviaguest If we do not get news, you can make a decision.

[ESSYCHAN]

My apologies Team, I'll run the experiments once more and give feedback this evening or latest tomorrow morning. I understand everyone has been busy trying to close the year, same for me here. It is hectic but pusing to close off the review as well.

Best regards, Esther.

On Tue, 7 Dec 2021, 22:27 Olivia Guest, @.***> wrote:

@D-Maurel https://github.com/D-Maurel @ESSYCHAN https://github.com/ESSYCHAN is there anything left unaddressed from Esther's feedback here? If not, @ESSYCHAN https://github.com/ESSYCHAN could you check things that have been made sense for you? Your feedback has been super valuable and I'd hate for it not be used to its full extent. ☺️

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ReScience/submissions/issues/48#issuecomment-988306356, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACH2RENNGZKP32PGGIG77UTUP2C33ANCNFSM4XM7MKKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[rougier]

@oliviaguest @ESSYCHAN Happy new year everyone. Any update on this?

[ESSYCHAN]

Hello and a happy new year to you!!

I have run in from my end, everything seems to be working fine on my end.

Additional comments (just one :) ) maybe add ** brew install cmake .. in the installation guidelines for those that don't have cmake

best wishes, Esther.

On Mon, Jan 3, 2022 at 9:04 AM Nicolas P. Rougier @.***> wrote:

@oliviaguest https://github.com/oliviaguest @ESSYCHAN https://github.com/ESSYCHAN Happy new year everyone. Any update on this?

— Reply to this email directly, view it on GitHub https://github.com/ReScience/submissions/issues/48#issuecomment-1003951302, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACH2REK3YK57ELLLF7CWUETUUFRCVANCNFSM4XM7MKKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[Kahss]

Hello and happy new year everyone :) !

Glad to hear that everything works well, thank you again @ESSYCHAN for your comments and feedback. The README has been updated with a pointer to a Stack Overflow post explaining how to run a Makefile in case you are on a Windows machine.

Best wishes, Denis Maurel

[rougier]

Cool. If @oliviaguest is ok, we can proceed with publication.

[Kahss]

Hi @rougier , @oliviaguest . Hope you're doing great. Do you have any update ?

[oliviaguest]

@D-Maurel Apologies for being so late about this, I keep getting unfortunate medical things happening. I think this is fine to process with publication. Has everything been filled in the metadata files (see: https://github.com/rescience/articles#readme) for the process to start? 😊

[Kahss]

I've updated the metadata file, and I've started to follow the process, but after having successfully reserved a DOI on the sandbox Zenodo server, I have a problem running the same command with the --zenodo flag :

> ./process.py --zenodo --metadata metadata.yaml --pdf article.pdf 
Traceback (most recent call last):                                                                                                                                                                                 
  File "./process.py", line 77, in <module>                                                                                                                                                                        
    article_id, article_doi = reserve_doi(server, token)                                                                                                                                                           
  File "./process.py", line 21, in reserve_doi                                                                                                                                                                     
    response.json()["message"])                                                                                                                                                                                    
  File "/home/xxx/.virtualenvs/chisel-replication/lib/python3.8/site-packages/requests/models.py", line 910, in json                                                                                 
    return complexjson.loads(self.text, **kwargs)                                                                                                                                                                  
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads                                                                                                                                                   
    return _default_decoder.decode(s)                                                                                                                                                                              
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode                                                                                                                                                   
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())                                                                                                                                                              
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode                                                                                                                                               
    raise JSONDecodeError("Expecting value", s, err.value) from None                                                                                                                                               
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 

Do you have any recommendation to continue ?

[Kahss]

Nevermind, after checking the request with Insomnia, it was just a problem of a account verification.

[Kahss]

My bad, as @ESSYCHAN has provided the link to the publication README, I thought I had to do it myself. I just stopped everything, and removed my post from the paper number claim issue. Everything is up to date in the metadata file :).

[Kahss]

Anyway, I hope you get better soon @ESSYCHAN !

[oliviaguest]

Oh, oops. I didn't mean for you to do the publication yourself. Just to check the metadata. 😅

I will get this published for you @D-Maurel, as soon as I can. Thanks for helping me to get it started.

[oliviaguest]

@D-Maurel I noticed you haven't updated the metadata in a while (11 months) or compiled the PDF? Sorry, I am rusty with this procedure, but I think you have to compile it for me? Also... @ReScience/editors are there any special requirements if this is on GitLab?

[Kahss]

Hi @oliviaguest , The PDF is two month old on the gitlab repository, could you please confirm me that this is what you are seeing ?

For the metadata, I thought I had filled everything that needed to. The part that is still empty is said to be filled by the editor. I assumed it was you but it was maybe a mistake ? Everything is complete except this part

# This information will be provided by the editor
dates:
  - received:  November 1, 2018
  - accepted:
  - published: 

# This information will be provided by the editor
article:
  - number: # Article number will be automatically assigned during publication
  - doi:    # DOI from Zenodo
  - url:    # Final PDF URL (Zenodo or rescience website?)

# This information will be provided by the editor
journal:
  - name:   "ReScience C"
  - issn:   2430-3658
  - volume: 4
  - issue:  1

[rougier]

You'll need to change the volume/issue as well. But the editor is mostly responsible for doing. @oliviaguest tell me if you need help on the publication.

[oliviaguest]

@rougier @Kahss I am so sorry to be taking so long. I will try to get this done today.

[Kahss]

@oliviaguest Absolutely no problem, I really hope that you're seeing the end of your medical things. Take care.

[oliviaguest]

@ESSYCHAN what is your orcid? @wmkouw I have found yours: https://orcid.org/0000-0002-0547-4817 😊

[ESSYCHAN]

Hi Olivia,

https://orcid.org/0000-0002-1327-1198

On Wed, Feb 23, 2022 at 8:20 AM Olivia Guest @.***> wrote:

@ESSYCHAN https://github.com/ESSYCHAN what is your orcid? @wmkouw https://github.com/wmkouw I have found yours: https://orcid.org/0000-0002-0547-4817 😊

— Reply to this email directly, view it on GitHub https://github.com/ReScience/submissions/issues/48#issuecomment-1048539026, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACH2REISNSRHGZGQKOQAPKTU4SKE3ANCNFSM4XM7MKKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[oliviaguest]

@ESSYCHAN thank you!

[ESSYCHAN]

Thanks Olivia. I hope you are feeling much better xx :)

On Wed, 23 Feb 2022 at 08:27, Olivia Guest @.***> wrote:

@ESSYCHAN https://github.com/ESSYCHAN thank you!

— Reply to this email directly, view it on GitHub https://github.com/ReScience/submissions/issues/48#issuecomment-1048543902, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACH2REOHRWVEPSGE6D3BO7TU4SK7BANCNFSM4XM7MKKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[oliviaguest]

@Kahss I think I now remember how to do this. 😅 Can you either allow me (same username as GitHub) to push to your repo on GitLab or update the metadata.yaml using this, please? And check everything is correct and then compile a new PDF for your paper called article.pdf?

# To be filled by the author(s) at the time of submission
# -------------------------------------------------------

# Title of the article:
#  - For a successful replication, it shoudl be prefixed with "[Re]"
#  - For a failed replication, it should be prefixed with "[¬Re]"
#  - For other article types, no instruction (but please, not too long)
title: "[Re] Effective Program Debloating via Reinforcement Learning"

# List of authors with name, orcid number, email and affiliation
# Affiliation "*" means contact author
authors:
  - name: Denis Maurel
    orcid: 0000-0001-5658-5923
    email: d.maurel@groupeonepoint.com
    affiliations: 1,*

  - name: Jérôme Fillioux
    orcid: 0000-0001-7304-1899
    email: j.fillioux@groupeonepoint.com
    affiliations: 1

  - name: Dan Gugenheim
    orcid: 0000-0003-4800-7909
    email: d.gugenheim@groupeonepoint.com
    affiliations: 1

# List of affiliations with code (corresponding to author affiliations), name
# and address. You can also use these affiliations to add text such as "Equal
# contributions" as name (with no address).
affiliations:
  - code:    1
    name:    onepoint sud-ouest R&D
    address: Bordeaux, France

# List of keywords (adding the programming language might be a good idea)
keywords: program debloating, reinforcement learning, unit test, python, machine learning, decision tree, delta debugging

# Code URL and DOI (url is mandatory for replication, doi after acceptance)
# You can get a DOI for your code from Zenodo,
#   see https://guides.github.com/activities/citable-code/
code:
  - url: https://gitlab.com/onepoint/recherche-et-developpement/optimus/chisel-replication
  - doi: 10.5281/zenodo.4530318

# Date URL and DOI (optional if no data)
data:
  - url:
  - doi:

# Information about the original article that has been replicated
replication:
 - cite: Heo, K., Lee, W., Pashakhanloo, P., & Naik, M. (2018, October).Effective program debloating via reinforcement learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (pp. 380-394).
 - bib:  heo2018effective
 - url:  https://www.cis.upenn.edu/~mhnaik/papers/chisel.pdf
 - doi:  10.1145/3243734.3243838

# Don't forget to surround abstract with double quotes
abstract: "This paper presents a replication of the CHISEL debloating system with some clarifications on caveats of the original method.
           It also presents a new property testing method using unit tests.
           The original results are extended with numerical proves of the improvements linked to the Reinforcement
           Learning part of the CHISEL system compared to the original HDD system.
           Several Python librairies are debloated and the improvements in terms of number of iterations and time consumption are presented.
           The results show that the RL part of CHISEL significantly improves the original HDD algorithm."

# Bibliography file (yours)
bibliography: bibliography.bib

# Type of the article
# Type can be:
#  * Editorial
#  * Letter
#  * Replication
type: Replication

# Scientific domain of the article (e.g. Computational Neuroscience)
#  (one domain only & try to be not overly specific)
domain: Program Debloating

# Coding language (main one only if several)
language: Python

# To be filled by the author(s) after acceptance
# -----------------------------------------------------------------------------

# For example, the URL of the GitHub issue where review actually occured
review:
  - url: https://github.com/ReScience/submissions/issues/48

contributors:
  - name: Olivia Guest
    orcid: 0000-0002-1891-0972
    role: editor
  - name: Wouter Kouw
    orcid: 0000-0002-0547-4817
    role: reviewer
  - name: Esther Mulwa
    orcid: 0000-0002-1327-1198
    role: reviewer

# This information will be provided by the editor
dates:
  - received: February 10, 2021
  - accepted: January 18, 2022
  - published: February 24, 2022

# This information will be provided by the editor
article:
  - number: 12 # Article number will be automatically assigned during publication
  - doi: 10.5281/zenodo.6255131 # DOI from Zenodo
  - url: https://zenodo.org/record/6255131/files/article.pdf # Final PDF URL (Zenodo or rescience website?)

# This information will be provided by the editor
journal:
  - name:   "ReScience C"
  - issn:   2430-3658
  - volume: 7
  - issue:  1

[oliviaguest]

@rougier I think I am doing this right, but please ignore/jump in as needed. 😆 Thanks. And sorry for being so rusty.

[Kahss]

@oliviaguest @rougier I've just updated the metadata.yaml file, however when I try to run the make command as mentioned in the README, the compilation ends with this error :

! LaTeX Error: File `biblatex.sty' not found.                                                                                                                                                                      

Type X to quit or <RETURN> to proceed,                                                                                                                                                                             
or enter new name. (Default extension: sty)                                                                                                                                                                        

Enter file name:                                                                                                                                                                                                   
! Emergency stop.                                                                                                                                                                                                  
<read *>                                                                                                                                                                                                           

l.308 \RequirePackage                                                                                                                                                                                              
                     {bibentry}^^M                                                                                                                                                                                 
No pages of output.                                                                                                                                                                                                
Transcript written on article.log.                                                                                                                                                                                 
Latexmk: Missing input file: 'biblatex.sty' from line                                                                                                                                                              
  '! LaTeX Error: File `biblatex.sty' not found.'                                                                                                                                                                  
Latexmk: 'pdflatex': source file 'biblatex.sty' doesn't exist. I'll try making it...                                                                                                                               
------------                                                                                                                                                                                                       
Running 'make "biblatex.sty"'                                                                                                                                                                                      
------------                                                                                                                                                                                                       
make[1] : on entre dans le répertoire « /home/d.maurel/.virtualenvs/chisel-replication »                                                                                                                  
make[1]: *** Aucune règle pour fabriquer la cible « biblatex.sty ». Arrêt.                                                                                                                                         
make[1] : on quitte le répertoire « /home/d.maurel/.virtualenvs/chisel-replication »                                                                                                                      
Latexmk: Errors, so I did not complete making targets                                                                                                                                                              
Collected error summary (may duplicate other messages):                                                                                                                                                            
  pdflatex: Command for 'pdflatex' gave return code 256                                                                                                                                                            
Latexmk: Use the -f option to force complete processing,                                                                                                                                                           
 unless error was exceeding maximum runs of latex/pdflatex.                                                                                                                                                        
Makefile:30: recipe for target 'article.pdf' failed                                                                                                                                                                
make: *** [article.pdf] Error 12

Maybe I'm missing something ? So far I've always used the overleaf compilation tool to get a pdf result, so I hadn't those problems.

[rougier]

You can try to type biber article on the command line.

[Kahss]

I've installed biber, run it and run make again, now I'm getting an article.pdf output, however I think there is still some problems as the received date is still "XX December 2020" and I get this output :

Output written on article.pdf (6 pages).
Transcript written on article.log.
Latexmk: Found input bbl file 'article.bbl'
Latexmk: Log file says output to 'article.pdf'
Latexmk: Found biber source file(s) [article.bcf bibliography.bib]
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 256
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs of latex/pdflatex.
Makefile:30: recipe for target 'article.pdf' failed
make: *** [article.pdf] Error 12

@rougier Any idea ?

[rougier]

Try a make clean, make, biber article and make again. If this does not work, you can try manual compilation.

[Kahss]

@rougier Ok so after investigation, there was a problem with the metadata.tex which was obsolete. Deleting it and running again make metadata.tex fixed the problem. There is a final version of article.pdf in the repository.

I was afraid there was an error regarding the insertion of a bib key just after "A partial replication of" in the article header, but when I compared to the latest published article, it seems to be the desired behavior. I didn't know if it was intended, so I wanted to mention it, just in case.

[rougier]

Well no, it is not the intended behavior. The bib key should be replaced with the actual citation. Which article did you check?

[Kahss]

Those ones:

• https://zenodo.org/record/5718075/files/article.pdf
• https://zenodo.org/record/5657320/files/article.pdf

[rougier]

AAAAaaaaahhhhhhh I'm among the co-authors, what a shame! So it's not supposed to be like this, I need to correct it. In your case, probably the simpler is to insert the full reference or to fix the script. But the citation should be processed by latex. Not sure why it is not.

[Kahss]

@rougier I've updated my pdf, everything should be ok. To solve the problem, I just had to change \replicationBIB line 94 and 95 in header.tex by \fullcite{\replicationBIB}. Hope it might help.

[rougier]

Cool. @oliviaguest I think the paper might be ready for publication.