eduble
commented
4 years ago Hi Paul
I'm trying to transfer files from walt nodes to the server. I want to retrieve experiment log files that can be pretty big that why I think using "walt-log-cat" is probably not the best option.
What I am searching for would be something like "walt node cp node_name:result.json ./" but from the client to the server (executed on the client). Is there such a tool ?
You probably mean "from the node to the server (executed on the node)"?
Another option could be to first receive filenames in logs on automatically transferring them to the server, but this involves some scripting server-side, so I would prefer avoiding that.
If I understood well, here is my point of view.
First, keep in mind that you will not be able to store very big files on a node. The node's filesystem is made of 2 layers:
- a read-only NFS layer corresponding to the selected walt image
- a writable overlay stored in the node's RAM
This setup has the following consequences:
- good: we can keep the SD card read-only. (Writing on it would greatly reduce its lifetime)
- good: the overlay being stored in RAM, it is discarded if node reboots. This ensures the node always boot exactly the same files (the ones of the walt image), thus it ensures a great level of reproducibility.
- bad: when writing big files, one may reach the maximum size of the overlay, which is not very large since nodes usually have little RAM amount.
We have a new feature which adds a directory /persist on each node. This is a read-write NFS share stored on the server, and specific to each node. You can store big files there. And they are not lost when node reboots. But I doubt this feature was ready with the version you installed. We will publish a new version in december or january that will include it.
There is a tool called walt-log-echo which may be used to send log lines one by one while the experiment script is running. This is probably the most effective way to use the logging system, when possible. (But if your output is not line-formatted, like in your case for instance, I understand why you tried walt-log-cat.)
For now I would suggest you use scp in your script on the node to transfer your files to the server.
Note that you can get the IP of the server by sourcing a script called walt-env.
root@vnode:~# . walt-env
root@vnode:~# echo $walt_server_ip
192.168.172.1
root@vnode:~#If you modify the node's image (walt image shell), generate a ssh keypair (using ssh-keygen) there, reboot the node with modified image, and copy the public key in file <user>@$walt_server_ip:.ssh/authorized_keys then you should be able to run scp without a password prompt (you will also need option -o StrictHostKeyChecking=no).
Warxo
Hello, I'm trying to transfer files from walt nodes to the server. I want to retrieve experiment log files that can be pretty big that why I think using "walt-log-cat" is probably not the best option.
What I am searching for would be something like "walt node cp node_name:result.json ./" but from the client to the server (executed on the client). Is there such a tool ?
Another option could be to first receive filenames in logs on automatically transferring them to the server, but this involves some scripting server-side, so I would prefer avoiding that.
Thanks, Paul.