Closed t00197857 closed 7 years ago
GALewis
commented
7 years ago That is correct. Our project does not use Dynamic VM Synthesis because it is not optimal for network-constrained environments which is the target of this work. VMs are pre-provisioned on the cloudlet, which also hosts the client to use the VM in case the mobile device does not have it (similar to an app store)
sebastian-echeverria
commented
7 years ago The code in this repository is not the code of our first prototypes. The first prototypes we developed did include VM Synthesis, and the references you mention may be talking about that version of the code. That implementation, however, was never open-sourced. The code included in this repo is of a newer version of the system that no longer contains VM Synthesis, for the reasons GALewis described above.
matthewparkhw
commented
7 years ago OK. Thank you for your reply.
Then, my next question is, which paper describes the architecture and technologies used in this project? And what's the relationship between this project and https://github.com/cmusatyalab/elijah-cloudlet.
What I'm doing now is to reproduce cloudlet demo (either one is ok) on our own testbed and explain the architecture and technologies (the more the better) that are used in it. So I need match the code and its corresponding papers. Would you please give me some suggestions about which project to use and which paper to study. Thank you.
matthewparkhw
commented
7 years ago BTW, I cannot find any code on VM manipulation techniques in pycloud project. The project is more like a mechanism that implement the process of cyber foraging, rather than the techniques described in the relevant papers.
If I want to study the code about the techniques, such as dynamic VM synthesis, baseVM creation, Overlay creation, Cloudlet-Ready Client App generation, Just-in-Time Provisioning, etc., then which project should I go?
sebastian-echeverria
commented
7 years ago Again, the code we have here is not the platform we developed for VM manipulation. This code is of a different project we created based on that one, this time focused on survivability. The first paper that refers to the code in this repo is "Cyber-Foraging for Improving Survivability of Mobile Systems" (http://ieeexplore.ieee.org/abstract/document/7357644/). The current version of the code in this repo, which has more security features, is referenced in the paper "Establishing Trusted Identities in Disconnected Environments", which was just presented and I think is not available online yet.
The code described in the papers "Cloudlet-Based Cyber-Foraging for Mobile Systems in Resource-Constrained Edge Environments", “Tactical Cloudlets: Moving Cloud Computing to the Edge” and “On-Demand VM Provisioning for Cloudlet-Based Cyber-Foraging in Resource-Constrained Environments”, is the previous platform we developed, which did have several VM manipulation techniques: VM Synthesis, On-Demand VM Provisioning, Application Virtualization, and others. Unfortunately, this code has not been open-sourced, so I can't point you to a link where you can review this code. The code you see in this repo is based on that one, but with all the VM manipulation stuff removed from it.
Finally, the Elijah project is a sort of brother project to the first platform we developed (the one with VM Synthesis, for which the code is not available). In that first platform we developed, we explored some of the same concepts that were being explored in the Elijah project, such as VM Synthesis. Both projects started with VM Synthesis at their cores, but from there diverged substantially. The Elijah project focused on optimizing VM Synthesis and exploring other issues, while we mostly started testing alternatives to VM Synthesis and comparing those to it.
So, in terms of papers and code:
sebastian-echeverria
commented
7 years ago And the paper "The Case for VM-based Cloudlets in Mobile Computing" references some code in the Elijah project repo, but I have no idea what version exactly. I guess the "Transient Customization of Mobile Computing Infrastructure" paper refers to code that may be in the elijah repo as well, but I am not sure where they actually put it.
Lincarl7236
commented
7 years ago Hello!Would you please give me some suggestions which repo should clone from the GitHub?
"Clone the repo from GitHub into a folder"
sebastian-echeverria
commented
7 years ago It refers to this repo, https://github.com/SEI-AMS/pycloud . Depending on what you want exactly, you may want to clone the latest version of the master branch of the repo, or you may want to download a tag. In the messages above there is a description of what each significantly different tag has.
Lincarl7236
commented
7 years ago OK.Thank you for your reply.
Lincarl7236
commented
7 years ago Execute the following commands:sudo bash start_dev_manager.sh There is a question "The 'adb==1.1.1' distribution was not found and is required by pycloud" . What should we do to solve this problem.
sebastian-echeverria
commented
7 years ago It sounds as if you did not set up the computer properly. Did you run the full_dev_setup.sh script as indicated in the instructions, and was it able to install all required packages? adb 1.1.1 is a Python package that is installed by the full_dev_setup.sh script.
If you ran the full_dev_setup.sh script as sudo, eg, "sudo bash full_dev_setup.sh", you may want to try again without sudo, eg, "bash full_dev_setup.sh". There may be some permission issues which may have lead to the packages not installing or not installing in the proper locations.
Lincarl7236
commented
7 years ago Based on your suggestions, I found that I did not set up the computer properly. There may be some permission issues lead to the question.When I ran the bash full_dev_setup.sh script .there are some questions，eg， "1. The directory '/home/lin/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. 2 Failed building wheel for netifaces. 3. Adduser: "kvm" group does not exist ,and so on. How can we solve these questions?
sebastian-echeverria
commented
7 years ago Btw, point 1 you mention there is only a warning, bu the rest look like errors. Let me check and I'll get back to you. What is the version of the host operating system you are installing this on? And what branch are you using, the latest version on master?
Lincarl7236
commented
7 years ago The system I used is Ubuntu version 14.10. And I follow the instructions in the "Development Environment Setup" to test running the pycloud server. Firstly,I clone the repo from GitHub,"https://github.com/SEI-AMS/pycloud". The path below is relative to the ./scripts folder .Secondly, I execute the command" bash full_dev_setup.sh". Then I log out from my Linux user, and log back in ，and to create the ./data folder. Finally ,I execute the commands from the root repo folder "sudo bash start_dev_manager.sh" and "sudo bash start_dev_api.sh". Is it right?
sebastian-echeverria
commented
7 years ago Yes, that is correct. Yesterday I tested those same instructions on a clean Ubuntu 14.04 machine, and other than having to create the data folder, I had no issues. I guess it is possible that Ubuntu 14.10 may have some differences that could create issues, but it is not what I would expect.
If you just followed that instructions again and you are still having issues, please reply back with an attached file that indicates the console output of the "bash full_dev_setup.sh" command, and of the "sudo bash start_dev_manager.sh" commands, and I will take a closer look at those logs.
Lincarl7236
commented
7 years ago What is the location of the data dolder? Today I test those instructions on a Ubuntu 14.04 machine,I do not find the data folder.So,I creat the data folder from the root repo folder, and I get a result ,eg,"Starting server in PID 2980.serving on http://127.0.0.1:9998" and"Starting server in PID 3149.serving on 0.0.0.0:9999 view at http://127.0.0.1:9999" .Is it right?
sebastian-echeverria
commented
7 years ago Yes, that means it is working properly! I guess there is some incompatibility with 14.10 then.
On Nov 26, 2016 05:28, "Lincarl7236" notifications@github.com wrote:
What is the location of the data dolder? Today I test those instructions on a Ubuntu 14.04 machine,I do not find the data folder.So,I creat the data folder from the root repo folder, and I get a result ,eg,"Starting server in PID 2980.serving on http://127.0.0.1:9998" and"Starting server in PID 3149.serving on 0.0.0.0:9999 view at http://127.0.0.1:9999" .Is it ringht?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SEI-AMS/pycloud/issues/6#issuecomment-263051360, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlUC52sYD0Puxw9e75SZmuSyamV3_a_ks5rB-21gaJpZM4K2BtF .
Lincarl7236
commented
7 years ago OK, thank you for your help.
Lincarl7236
commented
7 years ago Hello! KD-Cloudlet comprises a total of 7 GitHub projects: pycloud, cloudlet-client, client-lib-android, client-lib , android-logger, speech-server and speech-android. We can use the repo of pycloud to test running the pycloud server. But we do not know that what can we do with other repos. We want to set up a production cloudlet, and we have some questions, eg, "1. Is it necessary "Development Environment Setup" . 2. Where should we execute those commands in Production Setup. 3. Can we use pycloud in the case of a successful test？"
sebastian-echeverria
commented
7 years ago Lincarl7236
commented
7 years ago Thank you for your reply . I could not find the directory containing the deb package to execute the command "sudo dpkg --force-overwrite -i python-pycloud_0.2.0_all.deb" and I could not find the version number. Maybe my question is stupid, I am a novice, is very interested in this. I am sorry for the trouble I have caused you.
sebastian-echeverria
commented
7 years ago As I above, if you already set up the dev environment, you do NOT need to follow the instructions in the "2. Production Setup" section. Ignore that whole section, and its subsections, including the "Installation", "Post-Installation Files and Folders", and "Starting Pycloud". Those are only needed if you were setting up a long-term server. Just jump to section "3. Using Pycloud", and then follow the Service VM instructions I added above.
Lincarl7236
commented
7 years ago Thank you for your reply. When I create New Service VM, there is a question, eg, “There was a problem starting and connecting to service vm: Error creating Service VM: global name 'g' is not defined” . And I have no idea about how to generate the Disk Image file (.qcow2) and VM State Image file (.lqs). The Ubutntu 14.04 I used based on VMware Workstation.
sebastian-echeverria
commented
7 years ago Yes, there was a bug you just found which we had fixed in the working branch, but it was not fixed on the master branch. I just fixed it, so please pull from the repo to get this update. This should fix the "g not defined" error.
Regarding the creation of the files, the system will create them for you. The system will create the lqs file, so you don't have to worry about that one. You do have to supply a base disk image file, in form of a raw or qcow2 disk image, for the system to create the VM. In Ubuntu, you can create a new VM (and thus a new disk image which you can later use here) through virt-manager, for example. If you already have a VMWare disk image, you can convert it to qcow2 using qemu-img (google for that tool and that conversion and you should find examples of how to do it; the conversion is pretty simple with that tool).
Lincarl7236
commented
7 years ago I cloned the new repo and that problem has been solved . But when I create New Service VM, there is a question, eg,"There was a problem starting and connecting to service vm: Error creating Service VM: internal error: process exited while connecting to monitor: qemu-system-x86_64: -drive file=/home/lin/pycloud-master1/data/temp/servicevm/49345234-6fc5-4331-b1a2-84e4abc70187/lin100.qcow2,if=none,id=drive-ide0-0-0,format=qcow2: could not open disk image /home/lin/pycloud-master1/data/temp/servicevm/49345234-6fc5-4331-b1a2-84e4abc70187/lin100.qcow2: Could not open '/home/lin/pycloud-master1/data/temp/servicevm/49345234-6fc5-4331-b1a2-84e4abc70187/lin100.qcow2': Permission denied".
sebastian-echeverria
commented
7 years ago I can think of only two cases where that may happen: if the source disk image file you have has only read permissions, or if you are running cloudlet manager not as sudo. Are you sure you are starting cloudlet manager as "sudo bash start_dev_manager.sh"? And is your disk image file read-only?
Lincarl7236
commented
7 years ago No,my disk image file is not read-only. All users have read and write permissions.
sebastian-echeverria
commented
7 years ago Assuming you are running cloudlet manager as sudo, check the file /etc/libvirt/qemu.conf. Check the user and group settings there. If they are set to a user, try either commentting them out, or setting both to root. Also comment out dynamic_ownership if needed. Then stop and start the libvirt daemon (audio stop libvirt-bin, audio start libvirt-bin).
On Dec 19, 2016 03:35, "Lincarl7236" notifications@github.com wrote:
No,my disk image file is not read-only. All users have read and write permissions.
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/SEI-AMS/pycloud/issues/6#issuecomment-267892167, or mute the thread https://github.com/notifications/unsubscribe-auth/ABlUCyUsLtIeL8B2RJgUitPGjUU47QWSks5rJiWygaJpZM4K2BtF .
sebastian-echeverria
commented
7 years ago Btw, that should have been "(sudo stop libvirt-bin, sudo start libvirt-bin)".... "audio" was the autocorrector....
sebastian-echeverria
commented
7 years ago And btw, the issue you were seeing with the permissions was due to the setup script being out of date. It was setting the user and group in /etc/libvirt/qemu.conf to the current user, instead of "root". If you set both values to "root" (with quotes), and set dynamic_ownership to 1 just in case, and then stop and start the daemon, the permission issue should go away.
Lincarl7236
commented
7 years ago Thank you for your reply. I will try again.
mejing
commented
7 years ago hello! I plan to use Pycloud Cloudlet Manager to create a VM with speech-server on it. but when I create a new VM,I don't know which required files I shuld Transfer for the server to the VM.
sebastian-echeverria
commented
7 years ago Hello mejing, the page Service-VMs in this wiki has information about creating SVMs, and the Speech page here gives more information about how to create that particular SVM fo Speech. More specifically, it indicates that "The ./src/dist folder has all configuration and model files that need to be on the same folder as the speech-server executable for it to run properly."
However, I know it can still be a bit painful to set the SVM up manually. I can suggest you the following to create the SVM more easily: there is a script to automatically create an osV-based Speech SVM in the Speech Server repository. The "capstan_setup.sh" script will install capstan for you, which is a tool for creating osV-based SVMs. Once you have that setup, the "create_svm_disk_image.sh" script will create a packaged Speech SVM that you can import in Cloudlet Manager with the Import button/feature.
Try that; if it doesn't work out for some reason, I can try helping you out or eventually sharing a link to the resulting packaged SVM as well.
mejing
commented
7 years ago thank you for your reply! as your said,I tried that.but when I create a packaged Speech SVM,there are some errors as follows: create_svm_disk_image.sh: line 6: capstan: No command found Cp: can not get "/home/jing/.capstan/repository/speech_osv/speech_osv.qemu" file status (stat): no that file or directory
Does it mean that I did not finish capstan installed or some problem I didn't realized.
sebastian-echeverria
commented
7 years ago Yes, that sounds as if Capstan did not install properly. Maybe it depends on the OS and distribution you are running under? Check the capstan_setup.sh file, try running each command independently, to figure out where the issue is. Or check the Capstan page for alternate Capstan setup steps. I'll check if I can generate the SVM locally meanwhile, to share it with you if you are unable to install Capstan.
mejing
commented
7 years ago I checked the capstan_setup.sh file,and runned each command independently,and didn't report any errors.but when I runned create_svm_disk_image.sh ,it still reported: line 6: capstan: No command found.and I can find Capstan application in /home/jing/bin/.but I didn't know if ~/.capstan/repository is there when it was installed. besides I checked the Capstan page,and try to install Capstan follow https://github.com/cloudius-systems/capstan/wiki/Capstan-Installation,and then when I runned create_svm_disk_image.sh ,it still reported: line 6: capstan: No command found.
sebastian-echeverria
commented
7 years ago The issue is before the line with ~/.capstan/repository.... is in the line that states "capstan build ...", from what I see. Ignore the script for a second. Can you simply execute the "capstan" command on its own? Does that fail? ~/.capstan/repository is where capstan will store VMs after downloading/creating them.
mejing
commented
7 years ago yes,I simply execute the "capstan" command on its own.but it report：open Capstanfile: no such file or directory.
sebastian-echeverria
commented
7 years ago Well, that mean it is installed... if you run "capstan build
mejing
commented
7 years ago yes,I have downloaded it.and when I imported it in Cloudlet Manager,the terminal reported asfollows, Starting server in PID 2748. serving on http://127.0.0.1:9998 Starting SVM import Starting SVM import Importing file: /home/jing/edu.cmu.sei.ams.speech_rec_service_osv.csvm Exception while importing: "filename u'edu.cmu.sei.ams.speech_rec_service.qcow2.lqs' not found"
sebastian-echeverria
commented
7 years ago What version of pycloud/cloudlet manager are you using? Versions previous to 3.0.0, which is the same that is currently in the master branch, do not show that error messages. Older versions do require a lqs file that is not included in the packaged SVM you downloaded.
Lincarl7236
commented
7 years ago Hello! Can you give me some relevant papers? I want to know the design principles of KD-Cloudlet.
sebastian-echeverria
commented
7 years ago Hello, here are some relevant papers we've written along with the evolution of the platform. Do note that some of the earlier features (like On-Demand VM Provisioning) have been removed from the current platform, or are part of earlier versions of the platform which were not hosted in GitHub.
Hello。I studied the code of pycloud under https://github.com/SEI-AMS/pycloud. But I found that the Dynamic VM Synthesis approach is NOT implemented in this project. And I cannot find any descriptions about baseVM and launchVM in the project wiki pages. It seems that the code doesn’t match the process described in the two papers, Transient Customization of Mobile Computing Infrastructure-mobivirt2008 and The Case for VM-based Cloudlets in Mobile Computing. Am I right? I’m not so sure about this.