search

researchart / rose6icse

12 stars 71 forks source link

timemachine #126

Closed minkull closed 4 years ago

minkull commented 4 years ago

https://github.com/researchart/rose6icse/tree/master/submissions/available/timemachine

Contact Lists

Authors


\author{Zhen Dong}
\affiliation{National University of Singapore}

\author{Marcel Böhme}
\affiliation{Monash University, Australia}

\author{Lucia Cojocaru}
\affiliation{Politehnica University of Bucharest}

\author{Abhik Roychoudhury}
\affiliation{National University of Singapore}

applicable for the Reusable and Available

dmz-peerreview commented 4 years ago

Dear authors,

Thanks for your submission. Overall, you have invested major effort in the preparation of the package:

Thjat being said, it does currently not run failing with the error: 

E: Unable to locate package linux-headers-5.3.0-24-generic
E: Couldn't find any package by glob 'linux-headers-5.3.0-24-generic'
E: Couldn't find any package by regex 'linux-headers-5.3.0-24-generic'
The command '/bin/sh -c apt-get update && apt-get install -y linux-headers-$(uname -r) virtualbox-dkms virtualbox virtualbox-qt  python-pip openjdk-8-jdk git-core      && pip install pyvbox enum vbox-sdk uiautomator' returned a non-zero code: 100

Although well elaborated and providing all necessary information for reusing your tool, I was not able to run the docker file using the documentation. As it stands at the moment, it I cannot the badge.

Minor remark: Note the missing cd/TimeMachine after git clone.

timm commented 4 years ago

@zhendong2050 can you do a pull request to fix the above issue?

zhendong2050 commented 4 years ago

Dear Tim,

Thanks a lot for the reminding. I will check the issue.

Best regards Zhen

On Mon, Feb 17, 2020 at 8:47 AM Tim Menzies notifications@github.com wrote:

@zhendong2050 https://github.com/zhendong2050 can you do a pull request to fix the above issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/researchart/rose6icse/issues/126?email_source=notifications&email_token=AEDG7ZZL6UFMNTDROTN3CZDRDHNDFA5CNFSM4KN7SWUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4X76I#issuecomment-586776569, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDG7Z23UMA2PRKYBTGNDCLRDHNDFANCNFSM4KN7SWUA .

-- Zhen Dong Research Fellow National University of Singapore https://www.comp.nus.edu.sg/~dongz/

zhendong2050 commented 4 years ago

Dear authors,

Thanks for your submission. Overall, you have invested major effort in the preparation of the package:

  • It provides a docker container and specifies all dependencies,
  • it uses a proper license,
  • it provides a neat architectural overview as graphic,
  • it provides demo content (not tested), and
  • it also considers hardware requirements and time expectations.

Thjat being said, it does currently not run failing with the error:

E: Unable to locate package linux-headers-5.3.0-24-generic
E: Couldn't find any package by glob 'linux-headers-5.3.0-24-generic'
E: Couldn't find any package by regex 'linux-headers-5.3.0-24-generic'
The command '/bin/sh -c apt-get update && apt-get install -y linux-headers-$(uname -r) virtualbox-dkms virtualbox virtualbox-qt  python-pip openjdk-8-jdk git-core      && pip install pyvbox enum vbox-sdk uiautomator' returned a non-zero code: 100

Although well elaborated and providing all necessary information for reusing your tool, I was not able to run the docker file using the documentation. As it stands at the moment, it I cannot the badge.

Minor remark: Note the missing cd/TimeMachine after git clone.

Dear reviewer,

Thanks for your effort on the review. Can you check your Ubuntu version? It seems you are using Ubuntu 18.04 with the linux-headers-5.3.0-24-generic header, which is not found in Ubuntu 16.04. Our docker file is written for Ubuntu 16.04, as shown in the README.

Can you please try it on again on Ubuntu 16.04 and make sure a linux header is installed? If you have further issues, please let me know.

Bests Zhen

minkull commented 4 years ago

@data-track-reviewer looking forward to your reviews

dmz-peerreview commented 4 years ago

Dear authors (@zhendong2050),

Thanks for the reply! Yes, I am using Ubuntu 18.04. What I personally don't understand is the need for the dependency as I always thought the isolation from the host OS (and between other containers) is what speaks for Docker. Isn't it possible to deliver the required linux headers in a defined version by using the Docker file without any dependencies to the host OS?

Thanks for considering! (But maybe we wait for the pending review first)

zhendong2050 commented 4 years ago

Dear reviewer, Thanks for your suggestion! Technically, it is hard to do what you suggested.

Virtualbox used in our technique requires the virtual device, /dev/vboxdrv, which has to be loaded on the host machine. Docker only does isolation and cannot provide virtualization. So the virtual devices has to be passed into the container from the host machine. Thus, versions of the operating system and linux header should be consistent between the container and host machine. It has to be Ubuntu 16.04.

I have tested on Ubuntu 16.04 with different versions of linux-headers. It worked well. I am wondering if you have a physical machine with Ubuntu 16.04 available. If you have one, our docker file should work well.

Thanks for your effort. Bests --Zhen

zhendong2050 commented 4 years ago

@dmz-peerreview My explanation is upstairs. Looking forward to your comment. thanks!

dmz-peerreview commented 4 years ago

Dear @zhendong2050,

Thanks for the explanation! This seems to be tricky indeed and I don't have a physical installation of Ubuntu 16.04 at hands. I would propose to first wait for the open review before rechecking; in any case, the explanation was useful - thanks!

zhendong2050 commented 4 years ago

Dear @dmz-peerreview Thanks for your efforts!

data-track-reviewer commented 4 years ago

First of all, reading through this thread, I almost overlooked that we are discussing two badges.

Available: Leaning towards accept

Github is a good platform to share an artifact such as this. Conserving the current version with a specific DOI could be of value in ensuring that exactly the version used in the paper is conserved. It appears that this is evaluated differently by different reviewers, so unless the track chairs say differently, I consider this a nice to have.

Reusable: Leaning towards accept

Complete agree with @dmz-peerreview , this artifact is nicely represented. Unfortunately, I am also unable to fulfill the technical requirements. Should not Docker in principle provide a solution for this? Could for example a public image be shared?

zhendong2050 commented 4 years ago

Dear reviewers @dmz-peerreview @data-track-reviewer

Thanks for your comments. DOI: I have created a DOI for the current version and put the DOI image and link in README files. The pull request was submitted.

A full image: I think it is difficult. First of all, setup of Android apps testing is complex and "heavy" since it involves simulation or virtualization which needs a virtualization module in the Linux kernel. Docker cannot do isolation on this level based on its official docs.

Virtualbox image: Our technique uses virtualbox as an internal component. I am not really sure it will work if we put our technique in another virtualbox. At least, the performance will decrease.

Typically Android apps testing setup runs on a physical machine. Our setup targets at Ubuntu 16.04, which I think is reasonable since Ubuntu 18.04 is a little bit new and is not very suitable for such research work. I am quite surprised Ubuntu 18.04 is already so popular.

Our technique has been tested on our 4 servers. I really wish I could create an account on our servers for you guys. Thanks!

dmz-peerreview commented 4 years ago

Dear all, I am not sure how to continue here. The explanations make sense to me and I honestly think that we reached a level where the effort to set up that physical Ubuntu installation is not justified for the review. I further assume that calling in a third reviewer having that very environment configuration at hands is up to impossible. In light of the elaborate documentation and reasonable explanations, I myself am leaning, pragmatically, towards granting the benefit of the doubt and proposing acceptance (provided the hard constraints and a brief explanation are made explicit in the documentation.

What do you, @data-track-reviewer, think? Maybe the chairs could also quickly elaborate. Thanks!

timm commented 4 years ago

I agree with @dmz-peerreview that the reviewers have taken reasonable steps to address this issue. Yet issues remain.

FYO: we are having this issue with many artifacts- not everyone has every installation available to them.

In situations like this, I assert available is suitable (since, strictly speaking, it does not require demonstrable executables) but reusable cannot be judged.

zhendong2050 commented 4 years ago

Dear reviewers @dmz-peerreview @data-track-reviewer

Thanks all for the effort you made on our artifact reviewing.

I have built an image for Ubuntu 18.04 and written a new docker file, which was uploaded to the Github repo. Can you please give another try on your ubuntu 18.04? thank you!

dmz-peerreview commented 4 years ago

Hi,

Not sure if this is still valid as the issue is already closed, but gave it a try again and it works now (until you need an Android device, which I do not have at hand).

One last minor issue provided the chairs agree to keep this change in the review cycle; Could you please make the docker image to your docker container (zhendong2050/android-x86-7.1-ubuntu-18.04) also publicly available?

zhendong2050 commented 4 years ago

Dear @dmz-peerreview,

Thanks for running it again. Glad to know it worked as designed.

Our technique does not require an Android device. It runs android apps in a virtual machine in which the Android OS is installed. We configured the virtual machine with a headless model so that you do not be bothered with GUIs during testing.

Could you please make the docker image to your docker container (zhendong2050/android-x86-7.1-ubuntu-18.04) also publicly available?

Yes, I have pushed the zhendong2050/android-x86-7.1-ubuntu-18.04 image on Dockerhub, which is publicly available. Anybody can download the image with the following command.

docker pull zhendong2050/android-x86-7.1-ubuntu-18.04

Dear reviewers @timm @dmz-peerreview @data-track-reviewer I have uploaded another image which is targeted at Ubuntu 18.04 and it worked well based on what @dmz-peerreview said. I also made the image publicly available. Could you please consider our latest change in artifact evaluation?

Appreciate the effort all of you made.

timm commented 4 years ago

thanks for everyone's hard efforts on this one.

@dmz-peerreview now reviews the work is potentially reusable

so I think we are nearly ready to go reusable and available

BUT

@zhendong2050, you need to show me one thing first:

@zhendong2050 : over to you

zhendong2050 commented 4 years ago

Dear Timm @timm ,

Thanks for reopening this issue and considering our the latest change. The CURRENT details in the rose6icse points to the new DOCKER image. I have updated INSTALL instructions and created a pull request. Thank you!