search

tunapanda / provision

System for provisioning a new virtual machine with Tunapanda Edubuntu
7 stars 12 forks source link

Need a mechanism for deploying edX courses #30

Open usernamenumber opened 9 years ago

usernamenumber commented 9 years ago

We could do this manually for the initial test deployment if needed, but ideally we should have:

devalih commented 9 years ago

Importing course feature already there by edx:

cd /var/tmp
  git clone https://github.com/edx/edx-demo-course.git
  cd /edx/app/edxapp/edx-platform
  sudo -u www-data /edx/bin/python.edxapp ./manage.py cms --settings=aws import /edx/var/edxapp/data  /var/tmp/edx-demo-course

For video streaming, We should include a new role "media server" to host static files and videos under nginx, this should be faster than using django to deliver that.

I already have a script to download videos from youtube and update the MongoDB to point to the new location, I'll clean that and add a pull request soon.

devalih commented 9 years ago

Some questions:

Does --settings=aws has to do with the AWS service of Amazon? Please explain its use.

edX production setup is so dependent on some amazon services such as S3, and yes aws settings has some of this configurations. You can use other env files as you desire, check here

Where exactly is stored the Course data? What if we can store it on local hard drive?

edX stores course data on mongodb, so the import command imports the course from the local disk to mongo. or in the below case from github to local disk to mongodb.

Is it possible to have a more sophisticate course to test features like "Liner Algebra"? May a more sophisticated demo course be developed so we can test a more complete set of EDX features?

Yupe of course, you can use edX-Insider course (developed by master Piotr) it uses much more features than the demo course.

I know is not difficult on EDX Studio to create a course, but is it easy to include these sophisticated features?

You can fork it and add whatever you want to test, and yes I agree some of the advance problem types are somehow hard to create for the average TA, but it shouldn't be a problem to anybody with a some programing skills.

Thanks for your answers.

Thank you for your questions :)

iLearner commented 9 years ago

I modified the EDX installation (Lubuntu 13) so it creates the SD card which includes, besides Lubuntu 13, EDX also so you don't have to install it. I only replaced the folder tree that contains the OS files with the one that has a complete installation of EDX. This requires that we have this compressed "tar.gz" folder tree, which has 2 GB, some where in the internet to be used by the script. I am currently uploading it to my Google Drive account, but I don't know if this is the right way. Please give suggestions. I believe this installation is complete. The "Demo" and EDX Insider" courses seems to work find, also from the EDX studio, but i still need a way to confirm if "Linear Algebra" and the "Apparmor" Module are working.

With this Method you can create the SD Card and also an Image of it, as we have already both types of scripts. The size of the full Image, otherwise, takes 5 GB and is more tricky to make because of the empty space it has to have.

We can use this to do tests while we continue working on the Ansible-Docker Solution.

I also will try to have the Ubuntu 14 Installation version totally complete so we have two choices.

On Tue, Dec 23, 2014 at 1:17 PM, Ali notifications@github.com wrote:

Some questions:

Does --settings=aws has to do with the AWS service of Amazon? Please explain its use. edX production setup is so dependent on some amazon services such as S3, and yes aws settings has some of this configurations. You can use other env files as you desire, check here https://github.com/edx/edx-platform/tree/master/lms/envs

Where exactly is stored the Course data? What if we can store it on local hard drive? edX stores course data on mongodb, so the import command imports the course from the local disk to mongo. or in the below case from github to local disk to mongodb.

Is it possible to have a more sophisticate course to test features like "Liner Algebra"? May a more sophisticated demo course be developed so we can test a more complete set of EDX features? Yupe of course, you can use edX-Insider https://github.com/pmitros/edX-Insider course (developed by master Piotr) it uses much more features than the demo course.

I know is not difficult on EDX Studio to create a course, but is it easy to include these sophisticated features? You can fork it and add whatever you want to test, and yes I agree some of the advance problem types are somehow hard to create for the average TA, but it shouldn't be a problem to anybody with a some programing skills.

Thanks for your answers. Thank you for your questions :)

— Reply to this email directly or view it on GitHub https://github.com/tunapanda/provision/issues/30#issuecomment-67999479.

usernamenumber commented 9 years ago

Thanks to both of you for the work you've done on this!

@iLearner, when the image finishes uploading, please post the link here. Also, does it include the ansible playbooks for edX, or just the services they install?

iLearner commented 9 years ago

Here is the link: https://drive.google.com/file/d/0B2Ncx5bTTuA8UVFZRF9majdfS3M/view?usp=sharing On the SD script you have to use that file instead of "rootfs-part2.tar.gz". That's it. I tried to make it downloadable with Ubuntu "wget", but seems that it is not possible with Google Drive. If we can find some place where it can be done like that, it may be would be good.

It includes the Root folder tree as it is when the EDX installation completes by following my EDX Lubuntu 13 Installation procedure on https://github.com/iLearner/x2go/tree/Lubuntu-13

On Wed, Dec 24, 2014 at 1:02 PM, Brad Smith notifications@github.com wrote:

Thanks to both of you for the work you've done on this!

@iLearner https://github.com/iLearner, when the image finishes uploading, please post the link here. Also, does it include the ansible playbooks for edX, or just the services they install?

— Reply to this email directly or view it on GitHub https://github.com/tunapanda/provision/issues/30#issuecomment-68074117.

usernamenumber commented 9 years ago

I am downloading this (internet is not very good where I am right now), but won't be able to test it until I can go out and get a new SD card (brought one with me, but it turns out it's bad).

Once I have it downloaded, I can at least put the file up in a temporary location that can be accessed with wget, and I'll talk to the Tunapanda folks about hosting it at tunapanda.org.

usernamenumber commented 9 years ago

I am back home now and have a new SD card. I'm extracting the tarball, but have an issue/question:

I was expecting a filesystem image I would then dd onto the card, but this appears to be a regular file archive. Do I need to edit the scripts in your iLearner/x2go repo to use this archive and then do execute sd with them?

iLearner commented 9 years ago

It is not an image because in that case it will be at least 5 GB big which is the size of the uncompressed root folder tree that you get when completing the EDX installation on Ubuntu 13. Instead It contains only the compressed Root folder tree as a "tar.gz" file. So to make it work use this file instead of the "rootfs-part2.tar.gz" file on the "sd" script to create the SD card (You can take it from https://github.com/iLearner/x2go/tree/Lubuntu-13) , or on the script we modified later from there to get directly the SD Card Image.. The late would be at least 10 GB as we did the script and would be to big to manipulate. After replacing the text you can do "python execute sd /dev/sd?". don't forget to have also the "execute" and "sd_card_partitions.out" files in the same folder.

usernamenumber commented 9 years ago

I tried the following:

  1. Clone the Lubuntu-13 branch of iLeaner/x2go/
  2. Comment out the line in sd that wgets the rootfs-part2.tar.gz file
  3. Download the archive linked above and save it as rootfs-part2.tar.gz in the x2go dir
  4. Run python execute sd /dev/sdb

After unpacking the archive, execute fails with this:

sync
OK!

mkdir /tmp/sdd2/home/linaro/EDX
mkdir: cannot create directory `/tmp/sdd2/home/linaro/EDX': File exists
ERROR!

Nothing was mounted there before, and the SD card was empty, so I'm not sure what's happening. It's getting late over here, so I'll pick this up again tomorrow. Just posting now in case you see an obvious fix.

iLearner commented 9 years ago

Sorry for that, I forgot that for this it is not needed, because it is already created.

Copy EDX Required Installation Files to /home/linaro/EDX on SD

Card

mkdir /tmp/sdd2/home/linaro/EDX

cp README.md /tmp/sdd2/home/linaro/EDX/README.md

cp execute /tmp/sdd2/home/linaro/EDX/execute

cp update /tmp/sdd2/home/linaro/EDX/update

cp edxPRE /tmp/sdd2/home/linaro/EDX/edxPRE

cp fixes /tmp/sdd2/home/linaro/EDX/fixes

cp edx /tmp/sdd2/home/linaro/EDX/edx

cp text_replacements.xml

/tmp/sdd2/home/linaro/EDX/text_replacements.xml

cp mongo_tasks_main.yml

/tmp/sdd2/home/linaro/EDX/mongo_tasks_main.yml

cp mongo_defaults_main.yml

/tmp/sdd2/home/linaro/EDX/mongo_defaults_main.yml

but the SD is already created because you only missed this section (Unmounting and removing the SD directories and eject):

Unmount SD Card Partitions

sudo umount /tmp/sdd1 sudo umount /tmp/sdd2 sudo rmdir /tmp/sdd1 /tmp/sdd2

Make SD Card Safe to remove (Eject)

sudo eject ${card}

On Sat, Dec 27, 2014 at 10:25 PM, Brad Smith notifications@github.com wrote:

I tried the following:

  1. Checkout the Lubuntu-13 branch of iLeaner/x2go/
  2. Comment out the line in sd that wgets the rootfs-part2.tar.gz file
  3. Download the archive linked above and save it as rootfs-part2.tar.gz in the x2go dir
  4. Run python execute sd /dev/sdb

After unpacking the archive, execute fails with this:

sync OK!

mkdir /tmp/sdd2/home/linaro/EDX mkdir: cannot create directory `/tmp/sdd2/home/linaro/EDX': File exists ERROR!

Nothing was mounted there before, and the SD card was empty, so I'm not sure what's happening. It's getting late over here, so I'll pick this up again tomorrow. Just posting now in case you see an obvious fix.

— Reply to this email directly or view it on GitHub https://github.com/tunapanda/provision/issues/30#issuecomment-68199557.

iLearner commented 9 years ago

I sent too fast the last email. Just to clarify, for future use of the SD script you need to delete or comment the EDX Installation files' section, because, EDX is already Installed in the Root folder tree provided, and I left those files there.

Any way, the SD card should be create completely by now and you can try to load it on Cubietruck. In my experience it is always good idea to eject the SD card so to take it out properly. Some times when you don't do it properly, it may fail to load on Cubietruck and you need to redo the SD creation process.

On Sun, Dec 28, 2014 at 12:17 AM, Arturo Samanez asamanez@gmail.com wrote:

Sorry for that, I forgot that for this it is not needed, because it is already created.

Copy EDX Required Installation Files to /home/linaro/EDX on SD

Card

mkdir /tmp/sdd2/home/linaro/EDX

cp README.md /tmp/sdd2/home/linaro/EDX/README.md

cp execute /tmp/sdd2/home/linaro/EDX/execute

cp update /tmp/sdd2/home/linaro/EDX/update

cp edxPRE /tmp/sdd2/home/linaro/EDX/edxPRE

cp fixes /tmp/sdd2/home/linaro/EDX/fixes

cp edx /tmp/sdd2/home/linaro/EDX/edx

cp text_replacements.xml

/tmp/sdd2/home/linaro/EDX/text_replacements.xml

cp mongo_tasks_main.yml

/tmp/sdd2/home/linaro/EDX/mongo_tasks_main.yml

cp mongo_defaults_main.yml

/tmp/sdd2/home/linaro/EDX/mongo_defaults_main.yml

but the SD is already created because you only missed this section (Unmounting and removing the SD directories and eject):

Unmount SD Card Partitions

sudo umount /tmp/sdd1 sudo umount /tmp/sdd2 sudo rmdir /tmp/sdd1 /tmp/sdd2

Make SD Card Safe to remove (Eject)

sudo eject ${card}

On Sat, Dec 27, 2014 at 10:25 PM, Brad Smith notifications@github.com wrote:

I tried the following:

  1. Checkout the Lubuntu-13 branch of iLeaner/x2go/
  2. Comment out the line in sd that wgets the rootfs-part2.tar.gz file
  3. Download the archive linked above and save it as rootfs-part2.tar.gz in the x2go dir
  4. Run python execute sd /dev/sdb

After unpacking the archive, execute fails with this:

sync OK!

mkdir /tmp/sdd2/home/linaro/EDX mkdir: cannot create directory `/tmp/sdd2/home/linaro/EDX': File exists ERROR!

Nothing was mounted there before, and the SD card was empty, so I'm not sure what's happening. It's getting late over here, so I'll pick this up again tomorrow. Just posting now in case you see an obvious fix.

— Reply to this email directly or view it on GitHub https://github.com/tunapanda/provision/issues/30#issuecomment-68199557.

usernamenumber commented 9 years ago

During a recent meeting, @pmitros pointed us to some freely available edX courses provided by MIT:

And here are a couple of edX courses on more basic subjects, which they made by converting Moodle courses from moodleshare using their moodle2edx tool. These courses might not be as well-designed as the edX courses, but if the script works reliably, they might provide an easy way to include a broader range of topics with the initial deployment by converting other courses from moodleshare.

@devalih: Earlier you said that it would be pretty easy to put together a script for this. Do you think you could put something together for testing some time soon and use it to import these courses? One request if you do: please have your script download the videos to /usr/local/tunapanda/data/video. I've got a related feature I'm thinking about that could do some cool things with that.

usernamenumber commented 9 years ago

And here's an electronics course that may be a better fit than either of the first two MIT courses linked above. My understanding from this would be a good one to prioritize: http://groups.csail.mit.edu/mac/users/gjs/6.002x/

usernamenumber commented 9 years ago

Ping @devalih re the video migration script mentioned in https://github.com/tunapanda/provision/issues/30#issuecomment-67934342