How to handle PPC64 arch and how to use already installed Node.js ?

[trex58]

Hi, (I have already opened: https://issues.apache.org/jira/browse/AMBARI-7275 about my issue about Ambari.) We are porting Ambari on PPC64. And building Ambari on PPC64 leads to download x86_64 Node.js code. Bad. So, we'd like to change Ambari in 3 possible ways:

• do not download node/npm but use local version: this is useful when, like in PPC64 case for now, no site can deliver the version for this architecture
• configure it so that it downloads Node.js from a different place
• configure it so that it handles the architecture (either get different files from a same place, or use different places) Would it be possible to get some help from you ? Regards, Tony (Grenoble - France) (I visited Oslo in August 1990: nice place !)

[eirslett]

PPC64, that's something of an edge case... I'm not sure if there's an easy solution to that. Is it important that your version can be merged back to Ambari trunk? Or are you running a platform-specific fork?

Can you run the normal Linux distribution of Node.js (http://nodejs.org/download/) on PPC64? In that case, it should be possible to patch this plugin, so it downloads and uses that binary.

You could try copying your own node binary (built for ppc64) to these places:

./contrib/views/capacity-scheduler/node/node
./contrib/views/files/node/node
./contrib/views/jobs/node/node
./contrib/views/pig/node/node
./ambari-admin/node/node

which is where it looks for your binary (You will need to do that for the root path of every submodule that uses Node). If it's already there, it won't download Node.js again. If that works, you should commit the node binary into your repo (but obviously they wouldn't allow that if you were to merge it upstream). I guess you could try symlinking them to just one binary to save some space.

The project maintainers could choose to make the frontend build an optional Maven step (activated with -Pfrontend maybe), and commit brunch's build output into the repo. Or use the maven-exec-plugin for everything and require that all developers have node installed on their system.

[trex58]

Hi

There are PPC64BE (BE = Big Endian) and PPC64LE (Little Endian, like Linux on x86_64).

Based on cloc report (below), Ambari looks to not contain code dependent on the underlying architecture, except one debug C file. So, it looks to me that Ambari should not be dependent on the underlying architecture.

Our goals are: 1) short-term: enable Ambari to run on PPC64 (LE & BE) and thus manage a PPC64-specific GitHub. 2) long-term: propose patches/changes to the community so that - some day - the main stream version of Ambari will handle also PPC64 versions. So, it is not a fork.

Node.js has been ported on PPC64 architectures. See: https://github.com/andrewlow/node . And it runs quite perfectly well on my machines. However, these 2 PPC64 & PC64LE versions are not available on http://nodejs.org/download/ . No idea if they will ever be available there. However, I know that there are plans to make PPC64 architecture more commonly used by the Linux community. Maybe there will be some day some PPC64 place on the Web where to download packages for this architecture ; maybe.

We have noticed that Ambari downloads x86_64 code from the web, from different places on Ambari. Maybe downloading the required Node.js code in one place would be enough, and share it. And maybe Node.js should not be downloaded if it is already available on the machine (and not in Ambari directories). This last point would help a lot. And maybe some check of the architecture should be done: do not download A1 code for A2 architecture.

I also noticed that the downloaded code is downloaded each time we try to build the Ambari packages. That would be fine if the needed code could be downloaded only if it has not been already downloaded once. And the downloaded code should be removed when doing "mvn clean".

Thanks for the list of places where the node command should be replaced. We have tried another way: put the node code, and some other downloaded .js code, in one unique place, and change the pom.xml. We'll study also your proposal. On your side, probably that you should have only one path where to download Node.js (if not available on the machine), and use it from the different places where you need it, instead of having "node" code in 5 different places.

Hummm reading your explanation about "If it's already there, it won't download Node.js again", it is not what I think it is doing. However, I do not master how the pom.xml files for Ambari do work, and maybe I've misunderstood the traces of Maven.

We do not have developers. We are trying to help to have Hadoop packages aware of PPC64 architectures, so that, one day, one can download source code from the official repository, build it, and use it, on PPC64 like on x86_64. With no change. One day... First step is to have a PPC64-aware GitHub, with our specific changes for PPC64 (which may be compatible with other architectures, or not, like a specific work-around). See : https://github.com/ibmsoe .

Thanks ! Tony

---

Language files blank comment code

Javascript 760 20173 38035 159400 Java 1126 31107 41684 125815 Python 599 16001 13088 69924 XML 435 5390 7602 42282 CSS 11 2347 1462 17907 SQL 41 3105 2505 7988 Pascal 152 1477 4036 7026 Bourne Shell 136 1020 3206 5994 PHP 23 293 708 4732 C# 48 649 1299 3655 Ruby 54 202 761 1080 MSBuild scripts 6 1 134 615 HTML 32 31 85 598 Bourne Again Shell 14 101 270 513 Lisp 1 22 46 145 XAML 1 9 22 117 Perl 1 12 22 80 vim script 2 25 37 78 DOS Batch 2 13 16 73 make 2 14 4 61 C 1 14 9 55 ASP.Net 2 5 0 43

JSP 1 3 18 8

SUM: 3450 82014 115049 448189

[Jonath]

Ok, first of all, thanks for taking the time to address our problem !

Indeed, the nodjs version we are using is not available on the official repo at http://nodejs.org/download/.

What is weird though is that your plugin seems not to check for the target architecture, leading to maven downloading a x86_64 binary of nodejs. I suggest to add some portion of code to check for an exception like this case - and an appropriate error message rather than downloading an unappropriate binary as a handler.

I finally managed to create a workaround that overcomes the build problem by creating a directory at the project root with all the binaries we need in order to build (node, brunch, grunt, bower, etc.) - which is pretty close to what you proposed in your answer. In ambari, your plugin is actually used only to provide access to these for the maven-exec-plugin.

Thus, I totally removed it from the pom files and modified the working directory where the maven-exec-plugin checks for the binaries to my own repo directory, and it builds. Using a flag to make the use of your plugin optional is a neat idea we may put forward on a JIRA when eventually trying to merge our version of ambari with the official one.

So, for the moment, we may be able to manage a standalone version for ambari on ppc, even though we have problems with a target product being a x86_64 labelled rpm (which is another issue unrelated to your plugin).

[eirslett]

What I would do is build the project on a Linux box, tweak the .gitignore files and then commit everything in the dist directory (result of the brunch build) into git. Then, remove frontend-maven-plugin from the pom. Switch to ppc, pull from your repo and do a clean rebuild - this time without Node and brunch, but the already-compiled frontend assets are there in the repo. :-) And if that works - send a PR upstream and ask them to change their build process so that more systems are supported. (Maybe they could remove Node from the build altogether, or replace it with Java-based tooling like JAWR.)

Good luck!