ARM port of new Meteor version 1.3.x [finished]

[TomFreudenberg]

Today we are starting with a new ARM port of Meteor 1.3.x.

We are using the latest beta branch from Meteor repository.

[toverux]

Hello, any update on this ? :)

[TomFreudenberg]

Hi Morgan ( @toverux )

working currently on final parts. Maybe a first one can be published on weekend.

Thanks for interest Tom

[TomFreudenberg]

Hi all,

most of the port is done, we have already run some examples on our ARM environments. It seems that MDG is preparing currently a next upcoming 1.3.1 update. So I decided to wait for that to not have too much useless work on building the binaries.

Have a nice weekend Tom

[derwok]

Maybe a good idea. For me the current 1.3 is not working at all. Neither on an Ubuntu box, nor on my Mac OS X. I always get this error after "meteor update" from 1.2.1 to 1.3: https://github.com/meteor/meteor/issues/6628 @TomFreudenberg thanks for keeping your universal port alive! ;-)

[Floyd25]

Very good! Thank you very much!

[TomFreudenberg]

Today MDG finalized the release-1.3.1 and set up new branch release-1.3.2

I will just wait for some more diffs, seems to me that the 1.3.x has currently a moving target and still not finished. Its a pity for me that this was pushed as a new release just to meet some conference/talk dates.

[TomFreudenberg]

By now, the iso build for 1.3.1 are also ready from MDG, so all meteors are updated to release 1.3.1.

Checking the diff between branches release-1.3.1 and release-1.3.2 they are currently working on the iso builder. In case that those will touch our porting work we are waiting some few more days for MDG to complete.

[TomFreudenberg]

Right from yesterday MDG started with a new (secondary) for dev_bundler on release 1.3.2 ... I will check that but still waiting for some "final" 1.3 release before doing too much senseless work.

[derwok]

No hurry. But one info: the bug meteor#6628 that I mentioned above is gone with current meteor 1.3.1

[TomFreudenberg]

Hi Wolfram ( @derwok ), yeah won't hurry. Great to hear that your bug is solved. Today MDG marked first "rc" tag, so 1.3.2 seems to be not far away.

[TomFreudenberg]

Short update from today. MDG had marked the sources with 1.3.2-rc.4 tag. Hopefully it won't take too much updates until new 1.3.2 will be pushed.

[TomFreudenberg]

Having 1.3.2-rc.6 today

[Zaiban]

Well they just released the final 1.3.2. Hopefully get to test my app on ARM soon. Thanks for all the hard work!

[TomFreudenberg]

OMG ... after realeasing 1.3.2 now already on 1.3.2.1 ... Quality has to get better for stable release decisions ...

[TomFreudenberg]

And neither 1.3.2 nor 1.3.2.1 are yet available as official binary download ... meteor --version still on 1.3.1 ... However, we go through the final fork process now

[TomFreudenberg]

We got it nearly 90% ... looks fine as far.

I had a short request to forum about current status for 1.3.2.1 and got the info that there is a 1.3.2.2 coming shortly. I will put that also into our work.

Read more at meteor forum

[TomFreudenberg]

Looks like 1.3.2.2 is now marked as default and current stable ...

[toverux]

Stable, until MDG wants a 1.3.2.3 :D

[TPXP]

Anyway, it seems they are going shorten the interval between releases, aren't they?

You might want to release 1.3.2.2, I'm pretty sure there won't be breaking changes between these releases. But feel free to make it your way, we're not in a rush, at least I think so. ☺️

[TomFreudenberg]

Uh ... there was a miss on MDG side when preparing 1.3.2.2 ... so they revoked that release and are preparing a hot 1.3.2.3 ;-)

.....

edited: proceeding with 1.3.2.4

[NoahVlncrt]

what's the status on this? Any news?

[TomFreudenberg]

Hi Noah ( @NoahVlncrt )

thanks for your message and interest. We have nearly been finished our work and rewritten some parts. I guess we are ready to release 1.3.2.4 and this seems to be quite stable, so we do not wait for 1.3.3. But also building the binaries take its time. If I have to make a time expectation: within the next 10-14 days.

Cheers Tom

[rdagger]

Looking forward to your release. Any performance improvements on the Pi with 1.3?

[NoahVlncrt]

@TomFreudenberg thanks tom for getting back to me. I can't wait for the release :)

[TomFreudenberg]

@rdagger I have not worked directly on the Raspi hardware currently. We do porting and builds via scaleway servers because they are much faster than any other board I currently use.

But if you have some ideas for benchmarks please visit meteor forum thread and leave a comment.

Cheers Tom

[TPXP]

Just wanted to add a quick note about performance : 1.3 will feature this awesome change which will make your app faster to rebuild. :wink: So, yes, there will be some performance improvements with Meteor 1.3.

[NoahVlncrt]

Not trying to be a pest but exactly how does the process work of updating it to the latest version of meteor work? Is there a guide you based your original work on? I'm just wondering.

[TomFreudenberg]

Hi Noah ( @NoahVlncrt )

pest :-) nice wording, but no, you are not.

Updating releases from 1.2.? to 1.2.?+ was mostly done by cherry picking the necessary commits. After that I had to update the special repos for builtin mongodb and node. Mostly they were stable. Then just run the build process for the binaries and publish them. Last but not least push it as new default and inform everyone about a new release.

Since I also support xBSD I have to take care of some additional elements, like node-gyp where e.g. I also had to reach in a commit at google just to support Meteor on xBSD.

The 1.3.? release has changed a few parts of the scripts and the build processing. So some parts need to be re-coded to work. Thats I am currently on and also checking the actual devel branch to see that the upcoming releases could be made easier. In addtion there are the issues #45 #37 #40 which I have adressed to this new release.

I am sorry but it takes it time.

Cheers Tom

[NoahVlncrt]

Oh there is no problem I understand it takes time. I was more just curios on the process behind it all. Thank you for filling me in :)

[jhergeth]

Hello,any new development regarding 1.3.? ?

[TomFreudenberg]

Hi Joachim ( @jhergeth )

I am sorry, we were fully busy last 2 weeks on a customer project and have to finish this up to next weekend. I will right continue the work on work afterwards.

Thanks for understanding Tom

[RyanHoitt]

Anything needed help wise?

[NoahVlncrt]

I second @RyanHoitt is there anything we can do to help?

[linegel]

I third @RyanHoitt is there anything we can do to help?

[TPXP]

Hey there!

Here is my take on porting meteor 1.3.3.1 to ARM. It seems to work pretty well so far, so you might want to give it a try!

Prerequesites

This method relies on system-wide binaries of MongoDB, so you will have to install the MongoDB package provided by your distribution using your favorite package manager. Make sure the mongod and mongo binaries behave correctly (and are located in /usr/bin), and you're good to go !

Setup

First, grab the official meteor project, in version 1.3.3.1 :

git clone https://github.com/meteor/meteor.git --branch release/METEOR@1.3.3.1

Apply a small patch I made which fixes everything just for you. :wink:

curl https://gist.githubusercontent.com/TPXP/f2497b7861e3cc45e069cfee5ca37230/raw/5accf15df4e81a9b5b2951c7744f1da29d6b3d70/meteor_1331_arm.patch > meteor_1.3.3.1_arm.patch
cd meteor
git apply ../meteor_1.3.3.1_arm.patch

Then, generate the dev bundle as you would do on any supported system

./scripts/generate-dev-bundle.sh

And run meteor

./meteor --version

It works ! :tada:

Troubleshooting

If meteor reports an error similar to this one when starting an app :

=> Errors prevented startup:

   While checking for npm-bcrypt@0.8.6_1:
   error: No compatible binary build found for this package. Contact the package author
   and ask them to publish it for your platform.

You will have to copy the directory which corresponds to the faulty module from packages/non-core to packages (in the meteor install). In my case, I copied packages/non-core/npm-bcrypt and all its content to packages/npm-bcrypt. If you want to make sure to avoid this issue, just copy all these packages :

cp -R packages/non-core/* packages/

Of course, updating the package catalog takes a while as usual. But we now include a commit making sure updates do not occur too often (see #31 for more details).

[TomFreudenberg]

Hi Thomas ( @TPXP )

thanks for your Patch, this looks pretty near to a quick'n and fast but as you wrote ... still some errors left. Thats why I have not pushed anything yet. What kind of trouble do you have? Maybe we put our actions together.

Tom

[TPXP]

Hi Tom!

Thank you for this (amazingly) fast response ! I just revised my patch as I forgot a little check about architectures in the bundler, and now it works!

I just noticed that I still experience issue #14 but, as I said there, launching Mongo in parallel and giving meteor access to the database with the MONGO_URL setting seems to do the trick. I also have a strange issue with package usage report :

Failed to record package usage.
(This error is hidden when you are not running Meteor from a checkout.)
Error: Network error: wss://activity.meteor.com/websocket: unable to verify the first certificate
    at packages/ddp-client/stream_client_nodejs.js:184:1
    at packages/ddp-client/stream_client_nodejs.js:174:1
    at runWithEnvironment (packages/meteor/dynamics_nodejs.js:110:1)

But it does not seem to alter the behavior of the app, as it still starts. So in short it works! :smile:

If you want, we can put our efforts together, but that will be easier if you share your findings. :wink:

[TomFreudenberg]

Yes, as noticed, a lot of our patches to contributed modules and systems (like BSD) are not yet "in" your fast patch. So I will upload our current branch to do not get split on the fork. Does not make sense.

I will keep you informed and then we can have a look on it. So as you see we are also running the forks for https://github.com/4commerce-technologies-AG/mongo and https://github.com/4commerce-technologies-AG/node-fibers and https://github.com/4commerce-technologies-AG/node-gyp

Cheers Tom

[TomFreudenberg]

Hi Thomas ( @TPXP )

I pushed all parts to branch release-1.3.3.1-universal and currently building the binaries. So far I got the port ;-) thanks for pushing me again this time 👍

Two questions left:

1.) You reference to an OpenSSL Bug at https://gist.github.com/TPXP/f2497b7861e3cc45e069cfee5ca37230#file-meteor_1331_arm-patch-L24-L26

What is it for?

2.) I want to be as most as possible aligned to the official meteor release. So they are using node v0.10.45 I will provide the binaries and builder for the node releases as well.

So https://gist.github.com/TPXP/f2497b7861e3cc45e069cfee5ca37230#file-meteor_1331_arm-patch-L24-L26 is part of "nice" but not aligned to stable release from meteor.

Thanks for feedback of the OpenSSL bug Tom

[TPXP]

Hi Tom.

Thank you for taking the time to take a deep look into my patch, I was sure this part would catch your attention. :smile: As you can imagine, I did not put that here for the sole purpose of changing the things from the meteor project : I put it here because there was a need for it. Indeed, when I ran my tests with Node v4.4.5, I had this error message when attempting to run a project that was using accounts (through the accounts-password package, which depends on accounts-base).

=> Errors prevented startup:

   While selecting package versions:
   error: Package version not in catalog: accounts-base 1.1.2

   While refreshing package catalog to resolve previous errors:
   error: Network error: wss://packages.meteor.com/websocket:
   3069263872:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert
   internal error:../deps/openssl/openssl/ssl/s23_clnt.c:769:

This error means that meteor attempted to update the package catalog to find new versions of the packages I was using, but failed to connect to the server during the SSL handshake; which is handled by OpenSSL. As a consequence, it cannot get the packages I want to use, and the app doesn't start.

It took me ages to find the cause of the bug, and the only workaround I found around this was upgrading to the latest faye-websocket release, which fixes the bug. You can find a very simple reproduction case with this Gist (which uses meteor's version of the module). Just run these commands to get the error :

Get the files
$ git clone https://gist.github.com/6dda139332e1956a0ce42f774d9d61f4.git
Install the module
$ cd 6dda139332e1956a0ce42f774d9d61f4
$ npm install
Run the test
$ npm test
--> Displays an error while connecting
error Network error: wss://packages.meteor.com/websocket: 3069558784:error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:769:

close 1006

If you upgrade to the latest version of the package, you get the correct behavior:

$ npm install faye-websocket@0.11.0
$ npm test
--> Displays the response from the server, which complains about the fact that we do not respect the DDP protocol, but that's normal.
open
message {"server_id":"0"}
message {"msg":"error","reason":"Bad request"}

I have absolutely no idea why I have this bug, maybe it's because of NodeJS v4. If you don't experience it with my test case above on Node 0.10.45, then you don't need this part of the patch. But as I needed it on my side with NodeJS v4, I put it in my patch.

[TomFreudenberg]

Hi Thomas, thanks for that info, I will check that against the build of node v0.10.45 and keep you informed.

[TomFreudenberg]

Hi Thomas ( @TPXP )

after testing and building different apps and examples, I have not any kind of download issue like you have described above. So I guess it depends on your used node v4.x. As I support node v0.10.45 like meteor stable, I can't find any issue.

Would be nice if you could check again after I have uploaded the builds

Thanks Tom

P.S.: Your test from above post works without any problem on my node build v0.10.45

[TPXP]

Good to know, then it's a Node v4-related issue, that's one less patch we have to put. :smile: As a side note, the meteor team also upgraded the faye-websocket dependency for their release with Node v4 : https://github.com/meteor/meteor/pull/6921/commits/441ae4b90cc9c8876ff59f741ceedb75fb540d1e

[TomFreudenberg]

Hi Thomas ( @TPXP )

maybe we try to bring up an additional branch for v4 and involve some of the necessary patches. By now I would be happy to just have the 1.3.3.1 release running.

Cheers Tom

[TomFreudenberg]

When running meteor update on e.g. simple-todos example, I get follwoing error:

/home/meteor/meteor-1.3.3.1-universal/packages/promise/.npm/package/node_modules/meteor-promise/promise_server.js:165
      throw error;
            ^
TypeError: Cannot read property 'packages' of null
    at Command.main.registerCommand.name [as func] (/tools/cli/commands-packages.js:1721:12)
    at /tools/cli/main.js:1402:23

Any suggestion? @TPXP is it same on yours?

Just try please:

git clone https://github.com/meteor/simple-todos
cd simple-todos
meteor update

[TomFreudenberg]

P.S.: Also meteor update --packages-only will raise the same error.

[TomFreudenberg]

Ok, this seems to be an existing one. I checked it with the release 1.2.1. Same conditions:

/opt/meteor/dev_bundle/lib/node_modules/fibers/future.js:245
                        throw(ex);
                              ^
TypeError: Cannot read property 'packages' of null
    at Command.main.registerCommand.name [as func] (/tools/cli/commands-packages.js:1685:39)
    at /tools/cli/main.js:1378:23

Something I have to check but not necessarly yet.

[TomFreudenberg]

I opened an issue #49 for that

[TomFreudenberg]

Last but not least issue is "missing phantomjs binary for ARMs" #50

That will not allow currently to run the mocha test etc. and raise some errors on app they use that stuff like todos examples

[TomFreudenberg]

ALL RIGHT :-)

Pre-builts and release is now availalable. I have not marked this yet as the default branch, would like to get some feedback before doing that.

You can get that new meteor release via:

cd $HOME

git clone --branch release-1.3.3.1-universal --depth 1 https://github.com/4commerce-technologies-AG/meteor.git meteor-1.3.3.1-universal

$HOME/meteor-1.3.3.1-universal/meteor --version

The ARMv7l dev_bundle pre-builts already published on our bintray repos and will be downloaded during meteor installation automatically.

Thanks for some feedback. Tom