No restore for NodeRed instances with activated project function

[mickym2]

I do not understand why this is a problem of the Node Red adapter, but as requested - I create an issue here as well:

If someone uses the project function in a NodeRed instance nothing will be restored. As the flows are stored in the project subdirectory - and this folders remain empty - all flows are lost. All flows are lost. The complete project directory subtree needs to be restored, to have the local github repository back which contains the history of the projects.

In addition all other directories remain empty as well. lib directory for own flow library and data which are stored from specific node types and contain specific informations (like cronplus etc.)

All node_modules should be restored automatically by restoring a package.json with their dependencies and executing a local npm install. (see https://github.com/simatec/ioBroker.backitup/issues/730)

The corresponding issue in the backitup can be found here: https://github.com/simatec/ioBroker.backitup/issues/731

[Apollon77]

@mickym2 Please provide details to where what is stored ... I do not use node-red and also so do not know what is special when projects are used ... The problem also exists with the node_modules ... the default iobroker backup can only backup anything or nothing of a directory ...

[mickym2]

So if only all or nothing of a directory can be backuped - then the content of each node-red directory for each instance under iobroker-data should be backuped completely. If not all is backuped will lose functionality in each case. In general I am not a NodeRed developer, but I think I roughly understand the structure. So you have always a core installation with the core nodes and base functionality and then this is merged with one or more instances which can run in parallel. So the files of the core installation of the node-red adapter in the iobroker environment are in the node_modules directory under /opt/iobroker, the individual instances are then in the iobroker-data directory which contains also user defined node modules of either additional node types or any other node js library which can be installed and use in function nodes automatically.

So let me start with the explanation of the files and directories how I meanwhile understood NodeRed. I will start to explain first a node-red instance without the project functionality is switched on, which will be the case for 95% of users of the NodeRed adapter. In this case the flows.json is restored, which contains the whole logic and all flows, but the all additional nodes should be installed manually or placed in the adapter config - which is not practical and will cut meanwhile of the comfort of NodeRed to install those modules automatically.

The problem I see, when I restored with the backitup adapter not the whole directory is restored and is mixed with some standard files. In addition all restored directories after restore with backitup adapter were empty.

Unfortunately since version 2 of the NodeRed adapter there were made so adaptations to the javascript adapter which make things more complicated. So with version 2 of the node-red adapter it has been introduced to specify additional nodes and npm modules in the adapter config, but it is much more comfortable to either node-red install additional node libraries automatically or via the palette-manager. However after restore with backit-up, not the original package.json is restored- only a standard one without any dependencies for additional modules.

Part1: So let me first discuss the Node-Red directory with project function switched off.

So the hint the if the palette-manager is used additional nodes are not part of backup is a restriction which is in my opinion absolutely not necessary, if either the node_modules directory and the package.json is part of the backup. To save space of the backup it is only necessary to restore the package.json and execute a npm install in the instance directory of NodeRed. So I did this and everything was restored properly.

So when backitup restored a node-red instance directory - the files were restore and only empty directories and an empty package.json without any dependencies were found. { "name": "node-red-project", "description": "A Node-RED Project", "version": "0.0.1", "private": true } But the flows. json and the credentials were restore as well. I guess the config files are restored as well, as the contain the path to the core nodes of the NodeRed core installation part.

So let me explain the contents of the node-red instance directory without project function:

So the files were restored, but overwritten with a standard, empty package.json - as shown above. So the first thing which should be changed is that the original package.json will be backuped and restored.

The settings.js is overwritten with each start of the node-red instance so all instances of node-red with the adapter will share the same settings.

The config.user.json file contains only adaptations of the NodeRed editor, but becomes more important when the project function is activated. The config.node.json contains information to standard node types and are reference to the core part of the NodeRed installation.

The flows.json is the most important file as it contains all the logic and flows which are defined in this node-red instance. If project function is not activated it is restored by the backitup adapter. The flows.cred.json contains all credentials - which are encrypted when the project function is used.

Now it is a problem with the empty directories. The node_modules directory contains all modules and nodes which a user installed directly by the pallette-manager or automatically by node-red initiated to definitions within function nodes.

The lib directory contains flow libraries which are user defined and are lost - as they are not restored by the backitup adapter as well.

So all defined flows or flow fragments which are stored locally are lost as the lib directory is not backuped and restored.

In addition each node can create their own directory here to store configuration data which are not part of the flows. Some example in the next directory screenshot.

[mickym2]

Part2: So the Node-Red directory with project function switched on.

As I already restored the files manually - I do not exactly remember which files were exactly in the root directory of the node-red instance. But each directory in the node-red instance root directory (i.e. /opt/iobroker/iobroker-data/node-red) was empty.

So i am quite sure that the config.projects.json was not backuped are didn't contain the proper information. So I had to specify name and email for a local git repository again, which is part of the node-red project functionality. This file contains also an encryption password - which is used to encrypt the flows-cred. json. And encryption is mandatory when using projects.

So if this file with the encryption key is not present and you have not set explicitly an encryption key - you have to define and set a selfed defined password which is used for encryption.

If not - the flow itself is available (flows.json) but only an empty flows-cred.json. This means you have to define again all your credentials in each http-request node, email node, mqtt config node, sql config node etc. - in any case where user credentials are used. This took at least one hour as I had to modify about 60 nodes.

But even if you use the project function - you see - the node_modules directory remains in the root directory which means the additional nodes are shared between all projects.

But as you can see there is neither a flows.json nor a flows-cred.json in the root directory of the node-red instance. May be there could be a file - if you switch later to use the project functionality.

Everything is stored under the project directory and all subdirectories. As the backitup adapter does not backup and restore any subdirectory all my flows would be lost. So you can understand the title of the issue.

In addition you see other directories from different nodes which can optionally store configuration data - which are not part of the flows. (in this case cronplus node, homekit nodes etc. ) - So it is not a good idea - if someone think that there is always a defined structure of the node-red instance directory. As I said from beginning - easiest - backup the whole directory.

So now to the projects directory:

The project function creates a directory for each project.

The project directory itself contains then the flows. json and flows-cred.json

and the local git repository.

Within a NodeRed Project you can define branches and milestones which you are familiar as developer.

So you can define in your NodeRed project different branches and comittments

I haven't restored an intermediate state - as it looks complicated - but you can at least see all changes. ;)

I guess - i have explained the file structure as good as I know it - again I am a simple NodeRed user.

[Apollon77]

The problem is that the current backup mechanism can not correctly backup symlinks and such and they happen in node_modules ... so first of all we need ajs.controller enhancement to exclude content from backup and such ... so not that easy. So yes a"supporting solution" from backitup could help here till this is realized in controller

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within the next 7 days. Please check if the issue is still relevant in the most current version of the adapter and tell us. Also check that all relevant details, logs and reproduction steps are included and update them if needed. Thank you for your contributions. Dieses Problem wurde automatisch als veraltet markiert, da es in letzter Zeit keine Aktivitäten gab. Es wird geschlossen, wenn nicht innerhalb der nächsten 7 Tage weitere Aktivitäten stattfinden. Bitte überprüft, ob das Problem auch in der aktuellsten Version des Adapters noch relevant ist, und teilt uns dies mit. Überprüft auch, ob alle relevanten Details, Logs und Reproduktionsschritte enthalten sind bzw. aktualisiert diese. Vielen Dank für Eure Unterstützung.

[mickym2]

In my opinion it is fixed with separate NodeRed backup in backitup adapter.