Ideas for preventing blocking on feeds

[tresf]

Currently when the feeds are outdated they make a syncronous request on page load.

PHP likes synchronous activity and it is the default behavior, but I think this extra 500ms makes it appear the site is running slow, when it's actually waiting on a new JSON feed.

I think the feeds should instead read the JSON data from the local file system only, and then later, set some trigger/flag to grab a new version in the background.

A standard approach for something like this is crontab, where we do a wget every 20 minutes or so, but I don't see it available on tuxfamily.

Open for ideas.

[lukas-w]

Tuxfamily has cron jobs. https://panel.tuxfamily.org/?do=croncreate;group=lmms

[tresf]

If it is Web based that would explain how I missed it. (I was trying crontab -e)

[tresf]

It's a quick hack, but sometimes these are the best.

I introduced a new GET variable called max_age which can be invoked via URL. i.e: http://lmms.io/download?max_age=0

The commit was only a few lines and can be found here https://github.com/LMMS/lmms.io/commit/c9aa89da25592ad6c40a95cf90afb9f60214163c

Next, I bumped the page's internal expiration from 15 minutes to 120 minutes.

Last, I called wget http://lmms.io/download?max_age=0 > /dev/null on crontab every 15 minutes which forces a data refresh.

This should result in the a page which never expires but always loads quickly from a cache file.

I repeated this process for the community page as well. Please try them out and let me know. :)

[lukas-w]

It does load a lot quicker now, nice job :+1:

One idea for managing this in a better way: I'd propose a new folder structure for the repo.

┐
├── cron # private cron scripts that are regularly run by the server
│   ├── minutely # Community feed would go in this directory for example
│   ├── hourly
│   └── daily
├── htdocs # Public php scripts
│   ├── community
│   ┊┄┄ …
│   └── screenshots
┊

This allows us to put certain files under version control that should not be available to the public via the nginx server.

[tresf]

Sounds good. Feel free to change this anytime. We should probably move secrets into their own dir too so they are not mucking up the root(ish) directory. If we start using the cron for more stuff, this organization will make things easier to find.

PS, typo, 120 minutes not 120 seconds (fixed above)

Also if you find a way to feed in mailman and irc logs, please send them my way... :)

[tresf]

Oh I made crontab -e default to nano so it should be tremendously easy to edit (as vi/vim can be a nuisance for some people).

[lukas-w]

I just started doing this, so chances are that you won't see any changes on lmms.io for a while as I disabled the deploy script for the time being.

It may be possible that I don't get this finished today. Please excuse the off-time.

[tresf]

The forums are down. Fuck.

[Sti2nd]

They just worked? Have you seen Lucas' post, Tres?

[lukas-w]

Moved all public pages to a new subdir and re-enabled deployment. And who would have thought, this broke some feeds :snail:

[tresf]

I'm still not sure why... and this is the wrong bug report for this conversation.

If we need stuff out of public, we just move it up a directory. Can you explain this a bit better? Now I have to comb through and redocument the forums deployment wiki... I can probably fix the 500 error by fixing the relative link to the navbar etc but I'm still confused as to the benefit of having things this way...

[lukas-w]

I suggested this change (which is necessary for putting the cronscripts under version control) in my last comment in this very thread. You wrote:

Sounds good. Feel free to change this anytime.

Don't worry, and please don't touch anything! The forums are still there, I didn't delete anything. I just have not transferred them yet.

As I'll not be able to finish this today, I'll change the server to deliver the old deployment again.

[tresf]

Yeah I missed the part where the whole root file structure changed. The devil is in the details. You don't have to revert though. I can certainly work-around this, I've just been stopped in the middle of my work.

[lukas-w]

I've just been stopped in the middle of my work.

Sorry for the misunderstanding. Just for clarification: I did not touch /home/deploy/lmms.io/public_html in any way, so none of your work is lost. Instead, I created a new directory at /home/deploy/lmms.io/deploy for the new deployments, so that public php scripts are located at /home/deploy/lmms.io/deploy/public. After I did that, I changed nginx to serve that directory to see if things worked.

I just changed it back to the old document root so that you see nothing has changed there ;)

[tresf]

I honestly thought htdocs was a typo for public_html (and i didn't correct you since the world uses it so often). I just don't understand the benefit of moving our public files. I'm sure there's a technical reason for it, but I was under the assumption that cron was going to be side-by-side public_html... Any explanation would help here... :)

[tresf]

I changed nginx to serve that directory to see if things worked.

That makes much more sense. Your commit does use the word "move" but perhaps I don't understand ngix enough to know how it is handled by git.

[tresf]

Is this because our version control is deploying to the public directory? Couldn't we deploy to ~/lmms.io/ and ignore everything that's not named public_html and cron?

[lukas-w]

Now I'm confused as well. The website is now the old one again (before my commit), but is not receiving updates via automatic deployment. Unfortunately I don't have any more time now, and this change is bigger than expected, so I'll explain it to you tomorrow asap. Sorry again for the surprise commit.

Besides this, as I've seen your work regarding the forum now: You should not make changes to files in /home/deploy/lmms.io/public_html. Any changes will be lost when the next GitHub commit triggers, as the deploy script deletes all files in there prior to deployment.

[tresf]

Even stuff in .gitignore ?

[tresf]

I'm working from the /forums folder which isn't under version control.

P.S. Git doesn't show that you reverted your commit so that might be blocking it from coming down.

[lukas-w]

Even stuff in .gitignore ?

Yes, even stuff in .gitignore. I can change the deploy scripts' config to ignore certain files, which I, as I just noticed, already did for forum just for the case someone started migrating it. That's why your changes on the forum didn't get lost. Clever me.

Git doesn't show that you reverted your commit so that might be blocking it from coming down.

That's because I didn't revert my commit. Before pushing my commit, I changed the deploy target to a new directory other than /home/deploy/lmms.io/public_html so that this copy of the deployment remains safe and unchanged and nginx still serves the old website's version (the version before my commit). Therefore this directory and the website is not updated by GitHub pushes anymore.

Well, I'll backup the /forum to a safe place just in case.

No need for that, but I won't stop you if you'll feel better with a backup ;)

If there are still questions, I'll explain more as soon as I come home, in about half an hour.

[tresf]

I just noticed, already did for forum just for the case someone started migrating it.

Hmm... are you sure you did? https://github.com/LMMS/lmms.io/commit/ba22964dfc1de9c216027b53ebdad3e53a7b55c0 :bikini:

[tresf]

Yes, even stuff in .gitignore.

No need for that

lol @ u

[tresf]

I can change the deploy scripts' config

Ah, N/M, was still thinking git. Disregard the are you sure above. :)

[lukas-w]

The new deployment is now at http://lmms.io:8005 (with it's root at /home/deploy/lmms.io/deploy/public), the old one still at http://lmms.io:80 (with the old root at home/deploy/lmms.io/public_html) . I copied the forum over, it's working on both. So they're hardly different, except for the new screenshot order. I can still change the new root if you'd like, but that's just aesthetics.

Oh, and turns out that the YouTube feed is broken, the old deployment is still reading it from cache, that's why it seems to "work" there. I'll file an issue for that.

If things are still unclear, don't hesitate to ask me.

[tresf]

First of all, good work on getting this all going.

If things are still unclear, don't hesitate to ask me.

Well, I don't think you ever explained why we need to bump stuff into sub-folder among sub-folder. Your original diagram that I said "change anytime" was interpreted as a side-by-side of htdocs and cron. What I thought I was agreeing to was fixing the crontabs only. (I realize now I misinterpreted this as public_html and cron. However, what things have changed to now... well, I don't believe were every properly illustrated. Here's what I see now:

/home/deploy/lmms.io
┐
├── deploy
│   ├── cron
│   └── public
┊

I feel this is a lot of work to achieve version control on a few scripts and frankly, I'm going into folder overload as now we're burring lmms.io/deploy/public/community/index.php when at one point it was lmms.io/public_html/community.php.

I don't expect to be spoon-fed explanations, and I certainly appreciate this work but I'm still a bit surprised as to the direction. Why exactly can't we perform this sync on the base of /home/deploy/lmms.io and keep everything where it is?

Oh, and turns out that the YouTube feed is broken, the old deployment is still reading it from cache, that's why it seems to "work" there. I'll file an issue for that.

It's likely working just fine. I believe the team (@StakeoutPunch ?) deleted all the content so its getting no data and displaying no data :bird: I think there's still some question as to what content we put on YouTube in general and somewhere along the line it got cleared out.

[tresf]

@Lukas-W, Edit, I just read the cron bug report which illustrates you will be putting everything in lmms.io so if that's already underway, no need to explain that part.

[StakeoutPunch]

@tresf I can clarify that I removed all the content. As there were no videos it was nothing major. I didn't delete the playlist, as I am waiting on permission, but it is private.

We need to get specific people assigned to managing the YouTube page, so that everyone with access isn't arguing over how it should be done.

[tresf]

@StakeoutPunch no worries, this stuff will eventually settle. For now, I uploaded the OSX unfa-spoken demo video so I have something to code against. :thumbsup:

[lukas-w]

I'll start at the beginning. I hope you enjoy a little novel in the morning.

The point of all this

The original idea was to be able to put the cron scripts and other files (such as the secrets, as you just suggested in #44) under version control. Obviously, we don't want those files exposed through our webserver. However, our git repo's root used to be the document root that nginx served, i.e. we put all our public PHP scripts directly in there:

LMMS/lmms.io/
├── .gitignore
├── community/
┊┄┄ …
└── screenshots/
├── header.php
├── README.md
└

If it stayed like this, we could never put private files in there, because all of them would have been public and reachable via nginx (Just like some files that should not be but already are, see http://lmms.io/README.md or http://lmms.io/composer.json).

Keeping public and private files seperated in one git repo

So we had to change something about that in order to seperate public and private files inside the git repo, because this way they were all mixed up. The solution which we agreed upon was to move all public files in their own directory (I first called it htdocs, now it's called public in the repo) and have the rest stay out there (see my comment and your answer. Apparently there was a misunderstanding and it wasn't exactly clear to you what I was suggesting?). The repo would then look like this:

LMMS/lmms.io/
├── cron/ # private cron scripts that are regularly run by the server
│   ├── minutely/ # Community feed would go in this directory for example
│   ├── hourly/
│   └── daily/
├── public/ # Public php scripts
│   ├── community/
│   ┊┄┄ …
│   └── screenshots/
├── secrets/

How it looks like on the server

Now comes the part which I suppose is unclear: On the server, the files looked something like this:

/home/deploy
├── deploy.lmms/ # Deploy script served and triggered by GitHub via http://lmms.io:8010/
│   └── deploy.php
├── lmms.io/
│   ├── public_html/ # Public document root served at http://lmms.io/. The git repo's content was automatically being deployed into this directory
│   │   └ index.php
│   ├── GITHUB_CLIENT_ID
│   ├── GITHUB_CLIENT_SECRET
│   ├── update_feeds.sh

We would have wanted the cron dir to be side-by-side with /home/deploy/lmms.io/public_html on the server, like this:

/home/deploy/lmms.io/
├── public_html/
│   │   └ index.php
├── cron
│   ┊

This all would have taken these steps:

1. In the git repo: Move all public files to public_html subdir (just like described above)
2. On the server: Change the deployment script to deploy to /home/deploy/lmms.io.

However, I didn't do it that way. The reason is that there were quite some files in /home/deploy/lmms.io that you put there and that I didn't know. When changing the deploy target to this directory, all those files could have been deleted and/or overwritten during deployment. Preventing this would have caused a lot of work and would still have been quite risky.

What I finally did

Obviously I didn't want to just delete all your work there, so I decided to do it like this:

1. Disable automatic deployment, so that nothing in /home/deploy/lmms.io/public_html/ gets deleted when doing the next step (the forum would have been gone for instance).
2. Do the actual move in the git repo: Move all public files to a subdir, in this case it's called public. This step is the commit 6c64ec4dd9e2d25de2a0413d1032a9e72823caf0.

3. For safely testing out the new commit without breaking the old website in /home/deploy/lmms.io/public_html/, I set the new deploy target to a new isolated place, which is /home/deploy/lmms.io/deploy/, and re-enable deployment. I planned to then transfer all content in /home/deploy/lmms.io/ (which I didn't want to loose, as written in the last section) to the git repo. This is, as you can see above, the update_feeds.sh cron script and the secrets. The location /home/deploy/lmms.io/deploy/ has no meaning at all and can and likely will be changed.

   Due to this step, the old deploy target does not receive updates anymore.

4. To see if the deployment worked, I temporarily set nginx to serve the new deploy target's public dir /home/deploy/lmms.io/deploy/public/ instead of /home/deploy/lmms.io/public_html. Like ten seconds later, you noticed that you could not reach the forum anymore via http://lmms.io/forum, which is because the forum was manually added by you to /home/deploy/lmms.io/public_html and therefore is not part of the automatic deployment.

   However, I did not inform you about the nginx change, which I really planned to do for just half a minute to see if deployment worked (and more importantly, what still has to be done, which has turned to be the forum). Because of that, you understandably thought I had deleted the forum and with it all your work, not knowing that what you see on http://lmms.io/ was in fact a new document root. I apologize for this.

5. I instantly changed nginx back to serve the old website, so you could relax and see that the forum is still there. :bath:. Later in the evening, I set up nginx to serve both websites, but on different ports (80 and 8005), so that you still see the old website working, and I can continue working on the new deployment.
6. The plan is still to transfer things like the forum from the old to the new deployment, until everything in /home/deploy/lmms.io that is not part of one of the two deployments can safely be removed. After that, we can backup everything that's still not part of the git repo, and move everything to /home/deploy/lmms.io/ and the old dir structure will be restored again. ;)

I hope this answers most of your questions.

[tresf]

Thanks for the explanation. It all sounds good. :coffee: And I mean that this time. :hamster:

However, our git repo's root used to be the document root that nginx served, i.e. we put all our public PHP scripts directly in there:

LMMS/lmms.io/
├── .gitignore
├── community/
┊┄┄ …
└── screenshots/
├── header.php
├── README.md
└

Well, actually it was more like this but I'm sure that's just a typo.

LMMS/lmms.io/public_html
├── .gitignore
├── community/
┊┄┄ …
└── screenshots/
├── header.php
├── README.md
└

So as I understand it, we can push to the github repo and it will make it to the temporary destination (and temporary port) now, and the permanent location will be updated once you are comfortable with the changes?

If there's anything you need to assign to me, please do.

There's one consideration I'd like you to understand and that's that my machine can't run these cron jobs at all. This requires development to be done on the server itself. This means any new cron I create will first be placed above the cron folder, outside of the deployment area and then moved into it via version control once I'm happy with it. I don't mind this extra step, but I simply don't use Ubuntu enough on a daily basis to be able to justify cloning that crontab portion to my local machine.

[lukas-w]

the permanent location will be updated once you are comfortable with the changes?

Yep, kind of. Or the new location is gonna be the permanent one, I'll see if it's worth the effort.

my machine can't run these cron jobs at all.

This was one reason for me to even put the cron scripts into the git repo. You can just run them manually, there's no need to run them regularly on a local copy, or is there? I'll just add some run_all_cron.sh script to the repo and that's it.

[tresf]

You can just run them manually, there's no need to run them regularly on a local copy, or is there?

The point is when ti comes to creating new and changing shell scripts, I won't have the ability to test them, therefor I will author them outside of the cron folder until they are working, then check them into version control and add to the actual crontab file when I know they're working. I just didn't want this to come as a surprise as I won't be creating them blindly on my PC, I'll be creating them on the server.

[tresf]

Or the new location is gonna be the permanent one, I'll see if it's worth the effort.

I can't speak to the effort, but I'd pefer to not nest everything up a directory if we can avoid it.

Also if there's anything in lmms.io folder that needs a home or explaining, please let me know.

[lukas-w]

What prevents you from running the script(s) on your computer? With our only existing script, we'll just have to change wget http://lmms.io/community?max_age=0 -O /dev/null to wget http://localhost/community?max_age=0 -O /dev/null and it'll work on both local copies as well as on the server.

[tresf]

What prevents you from running the script(s) on your computer

A butterfly.

But you're speaking to a specific script (yes I can save the contents of a json file locally). The crons we create down the road won't necessarily be as easy to mimic. :beers:

[lukas-w]

Aaahh, I missed that part. That really is a handicap.

[lukas-w]

And if we'd rewrite the script in PHP?

[tresf]

I tried to, PHP on the command line breaks a bunch of stuff like DOCUMENT_ROOT doesn't even exist because there's no web server running.

But again, that's just what we're using scripts for now... we'll likely begin adding more stuff as time goes on and the likeliness of them all being wgets or php-able becomes more and more slim the more we want out of them.

It's fine. I don't mind doing it this way, but if in the future if you see the occasional shell script out of place, I can reassure you, I'll move it to its home shortly, if not, feel free to issue loud vulgar insults. :sailboat:

On a side note, I did buy one of the most expensive Ubuntu laptops money has to offer, but Ubuntu 12.04 was terrible on it and an OpenJDK bug keeps me from using it for daily coding use, so sadly even that is running Windows. I'll probably switch that once I can get my software to build without Oracle JDK. :wine_glass:

-Tres

[lukas-w]

if not, feel free to issue loud vulgar insults.

Haha, I'll keep that in mind :neckbeard:

I did buy one of the most expensive Ubuntu laptops money has to offer

There are Ubuntu laptops out there? I never looked for Linux compatibility before buying a laptop, and so far I've had not a single laptop or desktop Linux didn't run on.

Ubuntu 12.04 was terrible on it

If you're looking for alternatives, I can highly recommend Manjaro Linux (a distro based on Arch Linux), which is what I'm currently using and it looks like I'll be sticking with it. Like Arch itself, it uses a rolling release update model, which saves you from ever doing system upgrades again, but is way more user friendly and stable than Arch Linux.

a Java bug

Simple, purge Java from your system -> less bugs. :cocktail:

so sadly even that is running Windows.

Why not both?

[tresf]

Why not both?

Well, I guess you could say I do. I run VMs on all of my PCs (even the Mac) so I get a lot of time to be in Linux, OS X and even a few others like SteamOS, ReactOS, Solaris from time to time, however I find rebooting a computer to dramatically hinder production so I no longer dual-boot any of my systems.

Simple, purge Java from your system -> less bugs. :cocktail:

I wish I could but it's still paying some bills. :car:

But besides the terrible decisions Canonical has made with their UI, the reasons I can't use it on the laptop are due to the screen size and the hardware integration.

• Synaptec's driver on Windows detects whether my hand is resting on the trackpad versus using it as a mouse so I get much less accidental clicks. You'd think this would be minor, but it's actually detrimental when I'm managing files, editing with Gimp, etc.
• Multi-touch on Ubuntu was only possible using the Dell PPA and it still didn't work right. The laptop is a Mac-Air clone, so it has no mouse buttons. Multi-touch is used the entire time.
• Super high resolution display support on Ubuntu wasn't released until this year (the auto-scaling stuff to the likes of Mac and Win8) so I can actually read my code.

Those are it. I can live with just about any crappy desktop so as long as productivity prevails. :wrench:

[lukas-w]

Hm that's annoying, you would expect most of the things you're describing to work out of the box.

Seriously though, I suggest you take a look at Manjaro if you ever feel like giving it another try. The Arch Wiki is insanely comprehensive and I've never had any sort of problem the Arch Wiki did not contain a solution for. See the sections about your touchpad problem(?) or the page about HiDPI. Multi-touch worked out of the box for me in Manjaro (I suppose you're talking about the mouse pad, not the touchscreen, right?).

[tresf]

I suppose you're talking about the mouse pad, not the touchscreen, right?

Hmm... well, in Win8 multitouch works well on both the screen and the trackpad, although the touchscreen is a bit useless on a non-convertible laptop anyway...

Hm that's annoying, you would expect most of the things you're describing to work out of the box.

Well, they likely do in newer versions and I intend to switch back eventually.

Seriously though, I suggest you take a look at Manjaro if you ever feel like giving it another try. The Arch Wiki is insanely comprehensive and I've never had any sort of problem the Arch Wiki did not contain a solution for.

I'm a bit of a snob in that sense because I don't care to edit text files for hardware tweaking on productivity machines. I haven't had to do that on the other platforms for ages and I simply can't be productive when I'm adjusting synaptic config files. Also, I've been using Debian since Woody, so I've settled quite nicely with the structure and culture of it and Ubuntu is what I recommend to end users that ask to get off of Windows. It has (arguably) the largest user base and I also pay for desktop support through my printing company so that I can leverage the fixing of printing bugs. :dollar:

But there's no loyalty to any OS... I thoroughly enjoy the vast majority of OSs including the closed source onces. :crocodile:

[tresf]

And yes, I'll give that one a try.... :) :pacman: <--- No pacman? :)

[lukas-w]

Everything's back to normal again. The website is being deployed to /home/deploy/lmms.io, with the document root being at /home/deploy/lmms.io/public, served via http://lmms.io/. ;) Forum is there as well, and guess what, we even got new spam on it! http://lmms.io/forum/viewtopic.php?t=1416&p=5864#p5864

[tresf]

Fantastic, great news. Thanks for this.

P.S. Deleted the spam, blocked his email address. Thanks for the heads up.

[tresf]

@eagles051387 claims he has a fix to the spam. Jonathan, what access do you need to install this patch?

[eagles051387]

Ssh access as it makes some config modifications too On 29 Aug 2014 18:14, "Tres Finocchiaro" notifications@github.com wrote:

@eagles051387 https://github.com/eagles051387 claims he has a fix to the spam. Jonathan, access do you need to install this fix?

— Reply to this email directly or view it on GitHub https://github.com/LMMS/lmms.io/issues/5#issuecomment-53897173.

[lukas-w]

@eagles051387 Please describe the steps you are planning to take.

[lukas-w]

Oh, but please do that in issue #2. ;)