Ability to download image from url to be used for VM

[fcrozat]

This is a more a request for fbx-delta-nba_bash_api.sh than fbxvm-ctrl.

It is not possible to use fbx-delta-nba_bash_api.sh to do download API call to FreeboxOS

I have to use something like this: curl -s http://mafreebox.freebox.fr/api/v9/downloads/add -H "X-Fbx-App-Auth: $_SESSION_TOKEN" --data-urlencode 'download_url=https://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-MicroOS.aarch64-ContainerHost-OpenStack-Cloud.qcow2' --data-urlencode download_dir=${download_dir} --data-urlencode 'hash=https://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-MicroOS.aarch64-ContainerHost-OpenStack-Cloud.qcow2.sha256' --data-urlencode 'filename=delta-VM.qcow2'

It would be interesting to add a function to fbx-delta-nba_bash_api.sh to do this kind of api call (data-urlencode + several parameters).

[nbanb]

Yes, sure fbx-delta-nba_bash_api.sh need to have such function. There is another function I did not developed which is very close : the ability to download "Free pre-build Linux image" which should approximately work the same way (when I develop this tool, I was not interested first by this function because I'm using exclusively my own image with security hardening which match all ANSSI base requirements)

If I have a good memory, the way the download work on Freebox is :

• create a download task
• monitor this task using websocket API (or deprecated HTTP API)
• delete finished / failed task
• eventual use of file_system API to move, rename, cat, split ... the file

I think I can develop such function and extend it to Free pre-build Linux images at "lower time cost" because the vmdisk creation / resize workflow is working this way too, and I have already develop it in fbxvm-ctrl.

My only problem is I'm in FreeboxOS 4.6.5.1 and the download manager is not working (or/maybe I missed something) For example, with your curl :

19:42:20 nba@14RV-SERVER-37:/mnt/data/perso/freebox/api$ . fbx-delta-nba_bash_api.sh
19:42:25 nba@14RV-SERVER-37:/mnt/data/perso/freebox/api$ . loginfbx 
19:42:28 nba@14RV-SERVER-37:/mnt/data/perso/freebox/api$ download_dir='/FBX24T/dl'
19:43:19 nba@14RV-SERVER-37:/mnt/data/perso/freebox/api$ curl -s http://mafreebox.freebox.fr/api/v9/downloads/add -H "X-Fbx-App-Auth: $_SESSION_TOKEN" --data-urlencode 'download_url=https://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-MicroOS.aarch64-ContainerHost-OpenStack-Cloud.qcow2' --data-urlencode download_dir=${download_dir} --data-urlencode 'hash=https://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-MicroOS.aarch64-ContainerHost-OpenStack-Cloud.qcow2.sha256' --data-urlencode 'filename=delta-VM.qcow2' && echo

{"success":true,"result":{"id":8}}

The download task is sucessfully added in the download manager but after nothing happens ! (more that 1 hour that the task are in the download manager and nothing happens, no logs ...)

The direct curl from my laptop is working fine, so no troubles with the OpenSuze target server

I will try to reboot my freebox (need to wait, cannot proceed before midnight), but if it's still not OK, it would be difficult to blindly develop such function

[fcrozat]

My full script to do the download is more or less what you described:

        download_dir=$(echo -n "/Freebox/VMs"|base64)
        answer=$(curl -s http://mafreebox.freebox.fr/api/v9/downloads/add  -H "X-Fbx-App-Auth: $_SESSION_TOKEN" --data-urlencode 'download_url=https://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-MicroOS.aarch64-ContainerHost-OpenStack-Cloud.qcow2' --data-urlencode download_dir=${download_dir} --data-urlencode 'hash=https://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-MicroOS.aarch64-ContainerHost-OpenStack-Cloud.qcow2.sha256' --data-urlencode 'filename=delta-VM.qcow2')
  _check_success "$answer" || exit 1
  id=$(get_json_value_for_key "$answer" result.id)
  status=""
  while [ "$status" != "done" ]; do
        answer=$(call_freebox_api "/downloads/$id" )
        status=$(get_json_value_for_key "$answer" "result.status")
        echo $status
        sleep 2s
  done

The challenge was to encode some path in base64 and other in url-wwwencode (I had to do a lot of tests to find a working combinaison). Even if the path is in the wrong encoded, the download task is successfully added but will fail to download anything :(

BTW, you can have your own image built on openbuildservice for you (free advertising) ;)

[nbanb]

Thanks for your answer and this piece of script which solve the hard part of the job !

From the documentation, parameters to fulfill are :

 download_url (string) – The URL
download_url_list (string) – A list of URL separated by a new line delimiter (use download_url or download_url_list)
download_dir (string) – The download destination directory (optional: will use the configuration download_dir by default)
filename (string) – Override the name of the destination file. Only valid with one, non-recursive download_url.
hash (string) – Verify the hash of the downloaded file. The format is sha256:xxxxxx or sha512:xxxxxx; or the URL of a SHA256SUMS, SHA512SUMS, -CHECKSUM or .sha256 file. Only valid with one, non-recursive download_url.
recursive (bool) – If true the download will be recursive
username (string) – Auth username (optional)
password (string) – Auth password (optional)
archive_password (string) – The password required to extract downloaded content (only relevant for nzb)
cookies (string) – The http cookies (to be able to pass session cookies along with url). This is the content of the HTTP Cookie header, for example: cookie1=value1; cookie2=value2

Maybe all are not relevant or expected in all situations (for example, download_url_list , recursive ...)

As all physical path on Freebox hardware should be in base64 encoded, all others should be in url-wwwencode encoded, I think it could be easy to split parameters in 2 bash array of options, one for base64 encoded parameters and another for url-wwwencode parameters

I will see what could be done at the library part and at the fbxvm-ctrl script

Thanks for the idea, I will have a look on openbuildservice.

PS : I did find the issue with the download manager on my Freebox Delta ! if your interrested, have a look here (in French) https://dev.freebox.fr/bugs/index.php?do=details&task_id=37387

nbanba

[fcrozat]

As mentioned on the API doc, only mandatory is download_url or download_url_list, the rest is optional.

If you need help for OBS, just mail me ;)

(I ensured openSUSE image are playing nice with Delta and was able to get them listed by Freebox people ;)

[nbanb]

Dear Frederic Think I wrote the backend function for the lib : Is it what you expect ?

If ok for you, I will work on the frontend functions to put in fbxvm-ctrl (ex: dl-free-distro ; dl-custom-img ; dl-iso ...)

Let me know, and I can share the code ... Kind regards nbanba

[nbanb]

Dear Frédéric Sorry to be so late, I still didn't commit the new functions... Work is still in progress because I will release a pack of functions which are necessary to fully automated the creation of a sandbox and consumable VM in freebox delta environment. Functions will help in :

• downloading vm images
• managing vm images
• managing vm dhcp reservations
• managing incoming NAT redirection
• little file management.

I m also working on the ability to cache freebox json results when it reply an api LanHost object because almost json of this type are rarely less than 100kB and when applying several time function get_json_value_for_key, the listing process becomes too long and degrade user experience ...

I ll get back to you with the next big commit, Kind regards nbanba

[fcrozat]

Thanks for the notice, I didn't had time to reply to you, sorry. All of this sounds great and will probably make my script 95% shorter ;)

[nbanb]

Dear Frédéric Sorry to be so late again, I still didn't commit the new functions of the lib... Work is still in progress... But as I'm not developer , I didn't realized that what I announced to you about new functionalities (ability to friendly use freebox VM as a fully consumable sandbox VM) was equal to write 3 more "fbxvm-ctrl like" programs, one for managing downloads, one for managing filesystems, and one for managing networking functions which are for the moment: dhcp reservations and firewall redirection... There is another point: Having a deeper look on "fbxvm-ctrl" programs and it's fbx-delta_bash_api.sh library affinities, I realize that I didn't developped it in a modular and structural way as I mixed frontend and backend functions in it and in the library. Now, I did structurate (and comment as much as I can) the lib depending of which kind of functions and api are called by user or primitive programs, so I will have to rewrite "fbxvm-ctrl" nearly totally to use the new mindset of the lib... As soon as I will finished the filesystem tasks management in the library, I will release the new version of the library as a "pre-commit". This new version of the library include frontend autonomous functions for downloading vm images, making dhcp reservations or nat redirection and managing filesystem tasks. After this commit, I will work on the new version of "fbxvm-ctrl" including new functionalities and using library new mindset structure for frontend functions.

Kind regards Best wishes nbanba

[nbanb]

Dear Frédéric Sorry for the mistake, il will publish it not as a "pre-commit" but as a "pre-release" of the more globale work. As announced, I will publish the new version of the lib (with all new frontend functions) in the next 3 days. Hope you'll enjoy it !

There is something I would like to ask you: For the moment, I m the only testing peuple when I develop such functions, so my tests are influenced by the fact Im developping the functions and it's not a no real user feedback. If you can take a few minutes to test the new functions in your environnement when I will publish them, I would really appreciate your feedback, and if you cannot, really don't worry, I understand.

Thanks again for your help and feedback, Kind regards nbanba

[fcrozat]

No problem. Feel free to mail me directly to further discuss !

[nbanb]

Dear Frederic

I did commit the new version of the lib There were about +3000 lines of code, soplease download the new version and verify/modify the 6 configurables parameters of the lib

I did not write the README for the moment (big work), but the lib had been commented and all new frontend functions have an error output which show examples, etc...

Here is the commit description :

This is a huge commit, adding about 3000 lines of code to the previous version of this lib

In this commit, I'm adding several support and particulary fontend function wich have an output directly usable by an end user. The mindset of these new function is the same as "fbxvm-ctrl" program you can find on my public github repository
(https://github.com/nbanb/fbxvm-ctrl) New frontend functions are "autonomous" as they have their own "check param" function and their own "param error" functions. All "param error" functions can be configured to display the name of the program and arguments switchs of the program which is calling them. All new categories of frontend functions are also including a listing function, when necessary one monitor function with a nice auto scalable progress bar, a function which can print 1 task (not the list) and another function which give a "brut" result which can be parsed by a computer or another script. All new frontend function included error management and print error and detailed help when an error occurs in syntax or param and if error occurs at the API side (the API reply an error message), this message will be printed to the frontend standard output (stdout)

In this commit, I'm also :

_--> Adding support of FREEBOX_DEFAULT_URL and FREEBOX_LAN_URL and FREEBOX_WANURL _ - FREEBOX_WANURL preferred _ - FREEBOX_LAN_URL will be use if FREEBOX_WANURL is not defined _ - FREEBOX_DEFAULT_URL will be use if FREEBOX_WAN_URL and FREEBOX_LANURL are not defined

_--> Adding support of FREEBOX_DEFAULT_CACERT, FREEBOX_LAN_CACERT, FREEBOX_WANCACERT and _ FREEBOX_CABUNDLE which concatenate in a single CA certificate bundle all certificates of: _ - FREEBOX_DEFAULTCACERT _ - FREEBOX_LANCACERT _ - FREEBOX_WANCACERT

--> Adding support for ILIADBOX, the ITALIAN FREEBOX which had the same API --> Adding ITALY parameter support which will use ILIADBOXURL and ILIADBOXCACERT

_--> Adding support of ILIADBOX_DEFAULT_URL and ILIADBOX_LAN_URL and ILIADBOX_WANURL _ - ILIADBOX_WANURL preferred _ - ILIADBOX_LAN_URL will be use if ILIADBOX_WANURL is not defined _ - ILIADBOX_DEFAULT_URL will be use if ILIADBOX_WAN_URL and ILIADBOX_LANURL are not defined

_--> Adding support of ILIADBOX_DEFAULT_CACERT, ILIADBOX_LAN_CACERT, ILIADBOX_WAN_CACERT and ILIADBOX_CABUNDLE which concatenate in a single CA certificate bundle all certificates of: _ - ILIADBOX_DEFAULTCACERT _ - ILIADBOX_LANCACERT _ - ILIADBOX_WANCACERT

Some tasks (filesystem tasks, big download) can take hours and hours, really more than the login session timeout (~1800 seconds). So some of the frontend autonomous functions require a persistent login session. This part of the job is normally done by a frontend program which use functions from the library and ensure that the application still has a valid session opened. As librairy now contains frontend functions which can be used directly like an autonomous program by an end user, --> Adding support of auto re-login from librairy by library

OTHER IN THIS COMMIT : _--> Bug corrections of *BOX_CABUNDLE with websocat _--> fbx-delta-nba_bashapi.sh started to be BIG => structurating the librairy --> Adding comments to guide user configuration of library --> Adding comments for each groups of functions and for some functions --> Moving changelog at the end of the library for an easier configuration

####################### NEW FUNCTIONS ############################################

NEW CORE FUNCTIONS :

--> Adding core / call functions : - function for forcing a GET request with data-www-urlencode of parameters

--> Adding underlying functions for frontend functions : - function which colorize output depending on result - function which colorize json output depending on result and print json in "pretty-json" format to have a json human readable format

--> Adding underlying functions for testing network parameters validity - function to check mac address syntaxe - function to check ethernet port - function to check ip address syntaxe - function to check if ip is an rfc1918 ip address

--> Adding underlying functions for CA Certificate management - function which create a bundle CA Certificate file in shared memory - function which delete the bundle CA Certificate file in shared memory

--> Adding functions for managing autologin and auto-relogin in librairy : _ - function which publish _APP_ID and _APP_ENCRYPTEDTOKEN to subshell env at first login - function which logout the API - function which check the session status - function which get encrypted credential from environment and login with those credentials - function which re-login if the session is disconnected

NEW VM FUNCTIONS - FRONTEND :

--> Adding functions for managing Freebox VM prebuild distros: - function which list VM prebuild distro and export result to subshell - function which add and monitor download of VM prebuild distro - function which manage help / error and validate VM distro parameters

NEW NETWORK FUNCTIONS - FRONTEND :

--> Adding functions for managing DHCP static leases: - function which list DHCP static leases and usage status - function which add a DHCP static leases - function which modify a DHCP static leases - function which delete a DHCP static leases - function which manage help / error and validate DHCP parameters

--> Adding functions for managing incoming NAT redirection (WAN --> LAN): - function which list incoming NAT redirections - function which add an incoming NAT redirection - function which modify an incoming NAT redirection - function which delete an incoming NAT redirection - function which enable an incoming NAT redirection - function which disable an incoming NAT redirection - function which manage help / error and validate NAT redirection parameters

NEW FILESYSTEM FUNCTIONS - FRONTEND :

--> Adding functions for managing filesystem tasks: - function which list all filesystem tasks - function which modify a filesystem tasks - function which delete a filesystem tasks - function which show a particular filesystem tasks (pretty human readable output) - function which get a particular filesystem tasks (json output) - function which get a hash result on 'hash' filesystem action tasks - function which monitor a filesystem tasks (including progress bar) - function which manage help / error and validate filesystem task parameters

--> Adding functions for managing filesystem actions: _ - function list_fsfile: list content of a path / directory of freebox storage _ - function lsfs: CACHE & list content of a path on freebox storage ('ls' style) - function which copy a file/dir on freebox storage - function which move a file/dir on freebox storage - function which delete / remove a file/dir on freebox storage - f unction which rename a file/dir on freebox storage - function which create directory on freebox storage - function which hash a file of freebox storage (md5 sha1 sha256 sha512) - function which archive files or dir (.tar .zip .7z .tar.gz .tar.bz2 .tar.xz .iso .cpio) - function which extract archive on freebox storage (.tar .zip .7z .tar.gz .tar.bz2 .tar.xz .iso .cpio) - function which manage help / error and validate filesystem action parameters

DOWNLOAD FUNCTIONS - FRONTEND :

--> Adding functions for managing unauthentified share link (download links): - function which list all share link - function which add a share link - function which delete a share link - function which show a particular share link (pretty human readable output) - function which get a particular share link (json output) - function which manage help / error and validate share link task parameters

--> Adding functions for managing HTTP(S) / FTP download tasks: - function which list all download tasks - function which show a particular download task (pretty human readable output) - function which add a download task but do not urlencode params - function which add a download task and urlencode params _ - function which update a download task (iopriority, start, pause) - 2 functions which monitor a download task (scripting function & frontend advanced function) - function which print a download task log - function which delete a download task - function which manage help / error and validate download task parameters

--> Adding other download function : - function for making a direct download from Freebox storage (API authentified call, direction : server executing the direct downloadl function <------- freebox)

Enjoy and don't hesitate to ask for help ! nbanba

Here I'm also providing a complete list of all functions of the lib :

_add_dhcp_staticlease _add_dl_taskapi _add_freeboxapi _add_fwredir _add_sharelink _app_loginfreebox _archive_fsfile _authorizeapplication _call_freeboxapi _call_freeboxapi2 _call_freebox-wsapi _check_and_feed_dhcpparam _check_and_feed_dlparam _check_and_feed_fsparam _check_and_feed_fs_taskparam _check_and_feed_fw_redirparam _check_and_feed_share_linkparam _check_and_feed_vm_prebuild_distrosparam check_freeboxapi _check_ifip _check_ifmac _check_ifport _check_ifrfc1918 _check_loginfreebox checksuccess _checktool _check_toolexit _colorizeoutput _colorize_output_prettyjson _cp_fsfile _del_bundle_certfile _del_dhcp_staticlease _del_dl_taskapi _del_freeboxapi _del_fsfile _del_fstask _del_fwredir _del_sharelink _dis_fwredir _dl_task_logapi _dl_vm_prebuilddistros _dump_json_keysvalues _ena_fwredir _enc_dl_taskapi _extract_fsfile _full_vmdetail _get_freeboxapi _get_fstask _get_json_value_forkey _get_sharelink _hash_fsfile _hash_fstask _list_dhcp_staticlease _list_dl_taskapi _list_fsfile _list_fs_taskapi _list_fwredir _list_sharelink _list_vm_prebuilddistros _local_direct_dlapi _loginfbx _loginfbx2 _loginfreebox _logoutfreebox _lsfs _mk_bundle_certfile _mkdir_fsfile _mon_fs_taskapi _monitor_dl_task_advapi _monitor_dl_taskapi _mv_fsfile _param_dhcperr _param_downloaderr _param_fserr _param_fs_taskerr _param_fw_redirerr _param_share_linkerr _param_vm_prebuild_distroserr parse_and_cachejson parsearray parsejson parseobject parsevalue _print_termline progress _rebootfreebox _reloginfreebox _rename_fsfile _rm_fsfile _show_dl_taskapi _show_fstask _show_sharelink _statusfreebox _throw tokenizejson _update_freeboxapi _upd_dhcp_staticlease _upd_dl_taskapi _upd_fstask _upd_fwredir _vmresource wrprogress

Hope you can test and you'll enjoy !

Kind regards nbanba

Message ID: @.***>

[nbanb]

Dear Frédéric I m so sorry again, but as you suggest, I did reply by mail to your last comment and github website interface made a bad interpretation of the html/css of the mail... For a better reading and for having the real name of all listed functions, you should remove all '^_?' and all '_$' of the text of the previous comment and also remove certains '\n' of the output .

Maybe it would be better if I resend you this mail on another real mail address, just tell me... And ... if you dont want to post your personal email address on this github.com issue, just tell me and I will create a oneshot usage email address on one of my public domains and I will publish it here. Then, you can securely send me your mail and I will reply from my personal email address before destroying the mailbox created for this use. Don't hesitate to ask ! (really don't hesitate, it's less than 3 minutes for creating a temporary mailbox on one of my public mailserver...)

Kind regards nbanba

[fcrozat]

just mail me at ....

[nbanb]

mail sent ! if you don't get it, check your spam regards, nbanba

[nbanb]

Dear Frederic I did publish the full README documenting the library. Regards nbanba