[RE] Effective Program Debloating via Reinforcement Learning

[Kahss]

Original article: Effective Program Debloating via Reinforcement Learning

PDF URL: https://gitlab.com/onepoint/recherche-et-developpement/optimus/chisel-replication/-/blob/master/Replication_CHISEL.pdf Metadata URL: https://gitlab.com/onepoint/recherche-et-developpement/optimus/chisel-replication/-/blob/master/metadata.yaml Code URL: https://gitlab.com/onepoint/recherche-et-developpement/optimus/chisel-replication/-/tree/master

Scientific domain: Program Debloating and Machine Learning Programming language: Python Suggested editor: Visual Studio Code or VIM

[rougier]

Thanks for your submission, we'll assign an editor soon.

[rougier]

@otizonaizit @benoit-girard @oliviaguest @gdetor Can any of you edit this submission?

[oliviaguest]

Happy to take this on. Can somebody explain how we manage GitLab submissions?

[rougier]

Thanks. And for gitlab, nothing special I think. If accepted, we'll ask archiving on software heritage that can deal with gitlab.

[khinsen]

Nothing special indeed. See here for a GitLab-based submission that we already handled.

[oliviaguest]

@Kahsius do you have any ideas for potential reviewers? Feel free to drop their names, or usernames (but without the @, to avoid needless notifications for them) here — thanks! 😊

[Kahss]

Hello everyone, and thank you @oliviaguest for considering my submission :smile: ! Actually I don't have any ideas for potential reviewers as it is my first submission to ReScience. Do I have to give some names for my paper to be reviewed ?

[oliviaguest]

@D-Maurel no, you do not have to — but it would be very useful.

[rougier]

@D-Maurel you can have a look at the board to suggest reviewers: http://rescience.github.io/board/ @oliviaguest You can use notification to ask all reviewers at once

[oliviaguest]

@rougier I have already started asking people from my own contacts, for now.

[rougier]

I meant via @ ReScience / reviewers

[Kahss]

Thank you @rougier for the link. After checking the reviewers list, I would suggest people with experience in Reinforcement Learning, namely:

• Konstantinos Chatzilygeroudis
• Emmanuel Bengio
• Amy Zhang
• Xuedong Shang

I hope it might help.

[oliviaguest]

👋 @emmanuelle @costashatz would either of you two be willing to review this? Thanks! 😊

[costashatz]

@oliviaguest thank you for the invitation. Unfortunately, this is a very busy period and I will not have time to review this submission correctly. If you do not find reviewers before May, ping me again in May which is usually a calmer period. Thanks again and sorry for not being able to review the submission now.

[oliviaguest]

@amyzhang and @xuedong would either/both of you be interested and free to review this? ☺️

[oliviaguest]

@raar1 do you think you might be able to review this? 😊

[oliviaguest]

@D-Maurel I apologise this is taking so long — my impression here is that most people are not checking their notifications, but I am emailing them or at least trying to locate their email addresses too. No luck so far, sadly. I am hoping things will improve. 🤞

[Kahss]

@oliviaguest Thank you for taking care of it, I'm not in a rush :). I hope things will improve !

[oliviaguest]

Thank you, @wmkouw, for being a reviewer. 😊

[oliviaguest]

@D-Maurel do you have any other ideas for who might be a good person to ask? If yes, please feel free to give me their names (ideally without tagging them). ☺️

[rougier]

Any progress?

[oliviaguest]

I am trying my best — as this is not really my primary (or even secondary) area of research, I am trying all the avenues I can. If anybody has recommendations, that would be appreciated. Pandemic and things relating to it, seem to be making it harder for people to have the time; which is utterly understandable.

[oliviaguest]

Ah, also... FWIW @wmkouw has agreed, but I think I cannot "assign" them without them commenting on this issue?

[wmkouw]

Hi, sorry for not responding. I did promise Olivia that I would review this paper. I have not been very communicative because I just came back from birth leave and I'm swamped with work.

I told Olivia that I could make time in week 22 (May 31th until June 4th). Hope that's okay.

[Kahss]

Thank you @oliviaguest and @wmkouw for dealing with my article :slightly_smiling_face: !

[oliviaguest]

I think it would be ideal to have one more reviewer — and yes, 100% OK with me for you (@wmkouw) to take a reasonable amount of time given your constraints. 😊

[wmkouw]

Hi, a small update: I read the paper on Wednesday and briefly went through the paper it is reproducing. I just need to go through the code to complete my review, but I haven't been able to run the examples yet. When I generate local virtualenv directories and run pip, the dependencies are being installed in my conda env directory instead of the local ones. I need to figure out how to fix that, but I've had back-to-back meetings since Wednesday.

TL;DR My review isn't done yet; I will go through the code next week. Hope that's okay.

[oliviaguest]

When I generate local virtualenv directories and run pip, the dependencies are being installed in my conda env directory instead of the local ones. I need to figure out how to fix that, but I've had back-to-back meetings since Wednesday.

@D-Maurel can you please help @wmkouw (see my quote in this comment) to use your code, as this is central to reproducibility. Make the changes needed to code and documentation to help others too, etc., of course. Thanks! ☺️

[wmkouw]

Update: we had a family medical emergency and I've been at the hospital last week. I am back to work but I am swamped with teaching this week. I will finish the review next week.

I'm sorry. Things don't always go according to plan.

[Kahss]

@wmkouw Thank you for taking the time to read my article. I'm really sorry to hear about your medical emergency, I hope everything went fine. I will of course help you to solve any problem related to the execution of my code, please ask any question you have :) .

[oliviaguest]

@wmkouw I am so sorry to hear this — and, please, take your time. 🌸

[oliviaguest]

@ESSYCHAN has agreed to review this as well. ☺️

Esther, if you can drop a comment below, so I can assign you that would be great — thanks.

[wmkouw]

I've managed to find some time to finish my review. Instead of integrating conda en virtualenv, I just grabbed a Docker container and worked from there. Everything runs as advertised.

---

Summary: This papers successfully replicates an influential technique for program debloating. The original implementation was written in C, whereas this one is in Python. It includes tests for the debloater on Python code and reports additional quantitative results.

Strengths:

• A python version of the debloater is valuable and demonstrates the language-agnostic aspect of CHISEL.
• The paper discusses several key practical considerations that were not reported in the original paper, for example the requirement of multiple runs (Sec 2.3).
• The paper explores the use of existing unit tests instead of a test script that would have to be written prior to debloating.
• Table 2 adds an important perspective: the original paper reports large reductions in the number of statements (Figure 8) for CHISEL. However, this Table shows a less than 10% gain in process duration over no RL, which is much more modest.
• The paper is well written.

Weaknesses:

• Relying on UT can be tricky in practice, I think. In Section 2.2, it says: "However, this approach requires the software test suite to cover all the required functionalities." Suppose a co-worker leaves me with their large (and probably bloated) code base without telling me whether the tests have sufficient coverage. Should I just run the debloater? Or should I inspect the code thoroughly first, in case the debloater accidentally removes important functionality?

Minor points:

• (Sec. 2.3, RL) Perhaps worth adding: for gamma = 0, the RL agent defaults to a multi-armed bandit (performing 1 action and immediately observing reward; https://en.wikipedia.org/wiki/Multi-armed_bandit).
• (Sec. 3.2) ".. a while test suite". A "wide test suite"?
• (Conclusion) ".. outperform industrial debloating tools." I assume this refers to C-REDUCE and PERSES?

---

I apologize for my somewhat shallow review; I should have probed the code more thoroughly (I only ran the examples). It's been a trying time and this is all the effort I can spare at the moment.

[ESSYCHAN]

Hello @oliviaguest, Yes I have agreed that I'll review this paper.

Should I state the timeline on when I expect to finish?

[oliviaguest]

@ESSYCHAN that would be very useful and kind of you. 😊

[oliviaguest]

@wmkouw we appreciate your time and are very thankful. I hope things are as well as they can be. 🌼

[ESSYCHAN]

I'm committing to give a proper review by 19th July.

[Kahss]

Hello, I haven't noticed the notification mentionning that a review was available. I hope the delay hasn't done any harm, if not I'm really sorry.

Thank you @wmkouw for your review. Your point about the UT related weakness is totally relevant, and I will modify the original paper according to it, introducing further more nuances about the use of UT.

Thank you @ESSYCHAN for accepting to review my paper. I'm looking forward to read your comments.

[Kahss]

Hello all. I would like to know if you have any news ?

[oliviaguest]

@ESSYCHAN @wmkouw are your reviews possible to be done by the end of the month?

[rougier]

@oliviaguest Any update ?

[oliviaguest]

Nothing from me sadly... I tried 2 weeks ago as you see above. I will now try to email the reviwers.

[oliviaguest]

@D-Maurel have you addressed @wmkouw's comments and/or do you need feedback from them at the moment?

[Kahss]

@oliviaguest I haven't, I was waiting for another possible review to address every feedback at once.

[Kahss]

Hi all, @oliviaguest, would you have any update for a possible second review ?

[oliviaguest]

I'm really struggling to get anything from the other reviewer. Do you have any suggestions for a new 2nd reviewer? That might be the only way forward.

On Thu, 4 Nov 2021, 11:15 Denis Maurel, @.***> wrote:

Hi all, @oliviaguest https://github.com/oliviaguest, would you have any update for a possible second review ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ReScience/submissions/issues/48#issuecomment-960619910, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGYX3FIZ6MCGMTHKTXWD5TUKJMLFANCNFSM4XM7MKKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Kahss]

I don't sorry :/ . What is the minimum number of reviewer required ?

[ESSYCHAN]

Hello Denis,

I'm currently working on the review. should be able to give you feedback by close of business today.

Best wishes, Esther.

On Thu, Nov 4, 2021 at 10:45 AM Denis Maurel @.***> wrote:

I don't sorry :/ . What is the minimum number of reviewer required ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ReScience/submissions/issues/48#issuecomment-960640626, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACH2RELUDZH3IOOX6CS24QDUKJP5TANCNFSM4XM7MKKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Kahss]

Hello Esther,

Thank you very much ! I'm looking forward to reading your comments !

[ESSYCHAN]

Dear Maurel,

Attached, please find the PDF document with my comments on the paper:

Replication_CHISEL (1).pdf

I'll also issue a few comments here in this section:

I generally enjoyed reviewing your work, I think it is a commendable piece of work but there is room to add a few things.

The paper

It was an easy and straightforward read that gave me an overview about what has been done previously and what you currently are working on: A few things to note is

Clearly state your problem and how different it is from the previous implementation, I felt like there is more to do here and justify your work. Support most of the claims by expounding on your references. State as to why using Python as opposed to C (as previously done) would be beneficial for an RL task. This did not come out clearly (see comments in the paper) It is important to clarify about the objectives that you were working on, otherwise, it was hard to distinguish this from the replication. Is there a way to prove the quality of the UT cases (you will see comments in the pdf document) For the results, I suggested a few graphs for the reader to see the gain and improvement For the conclusion, give areas to work on in the future and a good abstract summarizing succinctly and expressively What the paper is about This comment was left out on the pdf about the first paragrph : (The introduction section needs to have several references. When you mention factors, it is important always to ascertain as to why such claims are being made)

Code

The code was a straightforward run other than struggling with setting up the Venv environment: The readme document should include details for different scenarios of V.env, this will help users not run into setup issues The test cases set up were easy to run given the commands, they all successfully passed s indicated by the results in the paper. I encountered a few errors while running the Optimus mainfile() on its own, although the test ran fine and successfully. File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/genericpath.py", line 42, in isdir st = os.stat(s) TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType I also noted that the test cases were resetting to range (3) on their own even after changing them to a different generator expression - You may need to check on this. the command $make example seems not to work properly Error received make: *** No rule to make target `example'. Stop. I particularly commend you on the code organization it was very clear to follow

With these few remarks and those on the pdf document, I wish you the best.

Let me know in case you need any clarification on my part.

Best wishes, Esther