Integrating a new simulation type into hpccloud

[robertsawko]

We are slightly struggling with the integration of a new type into hpccloud. I believe we found the right passages in the documentation, but we were not able to get it to work.

I have developed and compiled a new simput type it contains only model.js, convert.js and a few files in templates. I compile with my own version of Simput which is 2.2.20 and my colleague picks it up and puts to onto the on instance of hpccloud. From what I am seeing in the directory structure simput there is at version 1.5. Is that significant?

For the next steps we are looking in here. We're doing both of these things, but we get a blank page. We have also looked at the README.md as there are some instructions at the end there, but these seem to be incompatible?

So my main question now is what is the right way of adding new types? Do we put the source files and compile as part of the hpccloud build or can we generate the new_type.js file somewhere else? Do the version have to match? Do we need to edit components/steps/Input/index.js too?

[jourdain]

With the current UI of HPCCloud, you do need the version of simput to match. But you can still build the type outside of HPCCloud, you just need to install the right version via npm and use its cli.

For the integration inside HPCCloud itself, you should create a new directory here and make sure you register it inside the index.js of that directory. Then in your custom Input step you will have to make sure you reference the right type and possibly edit/decorate the simput model the way you need. You can start by just returning the model directly at first.

HTH,

Seb

[robertsawko]

This is, I believe, what we're doing apart from the version matching. Perhaps the way I am installing simput is wrong. Locally, I am trying this:

npm install simput@1.5.0

Then I point my env to bin directory and simput-cli.js -c src/ -o versions -t new_type. I get

/home/rsa/apps/simput/1.5.0/node_modules/webpack/lib/webpack.js:19
                throw new WebpackOptionsValidationError(webpackOptionsValidationErrors)

WebpackOptionsValidationError: Invalid configuration object. Webpack has been initialised using a configuration object that does not match the API schema.
 - configuration.output.path: The provided value "versions" is not an absolute path!

This seems strange but I provide absolute paths and get a different error:

%simput-cli.js -c $(realpath src) -o $(realpath versions) -t new_type
Error building                                                                                                                                                

/home/rsa/apps/simput/1.5.0/node_modules/simput/bin/compile.js:99              
            throw jsonStats.errors;                                    
            ^                                                 
Entry module not found: Error: Can't resolve 'expose' in '/home/rsa/projects/sandbox/simput/openfoam-tutorials'
resolve 'expose' in '/home/rsa/projects/sandbox/simput/openfoam-tutorials'

Also, there is a simput shipped with hpccloud. I guess we can simply use the one there, but we run into the same problem. There's obviously something about npm installation that we do not understand...

[jourdain]

You should install it globally so you will get a Simput executable in your path.

npm install -g simput@1.5.0

[robertsawko]

This may be pedantic, but I am using Lmod to manage my different software versions even on a laptop. Is there a way with npm to install into ~/apps/simput/1.5.0 with all dependencies?

[jourdain]

Ok looking at what you did previously the issue is actually that -o is to specify the path where the generated file will be located. Try to provide an absolute path that exist.

[robertsawko]

Yes, there's something very Sim-ple I must be doing very wrong...

Here's what I do:

% realpath versions 
/home/rsa/projects/sandbox/simput/openfoam-tutorials/versions
% simput-cli.js -c src -o $(realpath versions) -t openfuss
Error building

/home/rsa/apps/simput/1.5.0/node_modules/simput/bin/compile.js:99
            throw jsonStats.errors;
            ^
Entry module not found: Error: Can't resolve 'expose' in '/home/rsa/projects/sandbox/simput/openfoam-tutorials'                                               
resolve 'expose' in '/home/rsa/projects/sandbox/simput/openfoam-tutorials'
  Parsed request is a module
  No description file found
  resolve as module
    /home/rsa/projects/sandbox/simput/openfoam-tutorials/node_modules doesn't exist or is not a directory                                                     
    /home/rsa/projects/sandbox/simput/node_modules doesn't exist or is not a directory                                                                        
    /home/rsa/projects/sandbox/node_modules doesn't exist or is not a directory
    /home/rsa/projects/node_modules doesn't exist or is not a directory
    /home/rsa/node_modules doesn't exist or is not a directory
    /home/node_modules doesn't exist or is not a directory
    /node_modules doesn't exist or is not a directory

[robertsawko]

Thanks for quick responses and by the way:

simput-cli.js -V
1.5.0
which simput-cli.js
/home/rsa/apps/simput/1.5.0/node_modules/simput/bin/simput-cli.js

[jourdain]

Ok the deal is that simput aims to be installed globally so dependency will be found. Right now, it try to solve the dependency from your current path. A way to solve that is to maybe symlink /home/rsa/apps/simput/1.5.0/node_modules to ./node_modules

[robertsawko]

Yes, so we made some progress by editing bin/compile.js file adding -loader to both expose and babel. We are now receiving a very different errors. Something about missing presets

On a different note. Where do -g packages go?

simput-cli.js -c src -o $(realpath versions) -t openf
Error building 

/home/rsa/apps/simput/1.5.0/node_modules/simput/bin/compile.js:99
            throw jsonStats.errors;
            ^
/home/rsa/apps/simput/1.5.0/node_modules/babel-loader/lib?presets[]=react,presets[]=es2015!./src/index.js
Module build failed: Error: Couldn't find preset "es2015" relative to directory "/home/rsa/projects/sandbox/simput/openfoam-tutorials/src"
    at /home/rsa/apps/simput/1.5.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:293:19
    at Array.map (<anonymous>)
    at OptionManager.resolvePresets (/home/rsa/apps/simput/1.5.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:275:20)
    at OptionManager.mergePresets (/home/rsa/apps/simput/1.5.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:264:10)
    at OptionManager.mergeOptions (/home/rsa/apps/simput/1.5.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:249:14)
    at OptionManager.init (/home/rsa/apps/simput/1.5.0/node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
    at File.initOptions (/home/rsa/apps/simput/1.5.0/node_modules/babel-core/lib/transformation/file/index.js:212:65)
    at new File (/home/rsa/apps/simput/1.5.0/node_modules/babel-core/lib/transformation/file/index.js:135:24)
    at Pipeline.transform (/home/rsa/apps/simput/1.5.0/node_modules/babel-core/lib/transformation/pipeline.js:46:16)
    at transpile (/home/rsa/apps/simput/1.5.0/node_modules/babel-loader/lib/index.js:50:20)
 @ ./src/index.js-exposed 3:54-177

[jourdain]

Obviously you are hitting the next missing dependency that simput should have in its node_modules.

Regarding the global install that probably depend on the system and also their might be a way to tune it with some user config file. Look at npm for more details.

[robertsawko]

Ok, thanks. With npm, something that's not clear to me is the relation between modules and package I am installing. I had to install npm install babel-cli babel-preset-es2015.

There were some more missing *-loader commands in ~/bin/compile.json, @carpemonf. They're all around a line 37. In particular I added a few and it looks like this:

                      loader: 'expose-loader?Simput.types.' + name,
                    },
                    { test: /\.(png|jpg)$/, loader: 'url-loader?limit=8192'},
                    { test: /\.css$/, loader: "style!css!autoprefixer?browsers=last 2 version" },
                    { test: /\.json$/i, loader: 'json-loader'},
                    { test: /\.jade$/i, loader: 'jade-loader'},
                    { test: /\.html$/, loader: 'html-loader' },
                    { test: /\.hbs$/i, loader: 'handlebars-loader'},
                    { test: /\.js$/i, exclude: /node_modules/, loader: "babel-loader?presets[]=react,presets[]=es2015" },

[jourdain]

Try using 1.6.1 instead, I'm wondering if I fixed those between 1.5 and 1.6.

[carpemonf-zz]

Hi,

Thanks @jourdain, it worked for me without errors when switching simput to version 1.6.1 (for HPCCloud v2.0.0).

Regards.