search

Qubes-Community / Contents

Community documentation, code, links to third-party resources, ... See the issues and pull requests for pending content. Contributions are welcome !
257 stars 98 forks source link

Turtl Notebook-Manager (Client+Agent) in single offline AppVM #27

Closed Aekez closed 4 years ago

Aekez commented 6 years ago

All content here has been moved here due to only being partly on-topic to the 100% clean Browser issue at https://github.com/Qubes-Community/Contents/issues/24 and because it may be too detailed to keep in the same doc. So a separate doc may be recommended. Currently there is a missing component in the instructions to make the server persistent, as well as untested RAM performance. Other than that, this works in brief initial tests. I still need to finish up the missing parts, and do some more testing. Also a script to backup the server-data on a frequent basis is needed.

This post is a sketch, providing barebone instructions still falling short of the broader conception and goal of this post. Right now it's barely enough to get it working, it'll work, but nothing beyond that is documented or tested. What is being aimed for is instructions on how to make productivity and reliability, as high in quality far as possible, while also keeping it offline and private. Considering the barebone state of the post, this goal has not been reached yet. Be careful if you use this for sensitive and important data.

Sketch for doc starts here:

Quick summary sketch of below concept

Instructions (Once finished, to replace the old instructions further below)

Server steps (can optionally be skipped if you're ok with existing online server). It is worth mentioning that you can also use a hybrid solution, still following these steps to make your own server, but putting it on another computer acting as your own private server. It is however still exposed to hacking like any other solution if online.

Template preparation Please use this template for both constructing build AppVM and the later offline AppVM.

Using Build AppVM to build everything due to requiring internet access to build, and easier to clean up afterwards by simply deleting AppVM. Once building is finished, instructions recommend moving everything to an offline AppVM for use.

Client steps Using Build AppVM:

Using it

Backup steps

Planned To-do list for above doc:


. . . . . . . . . . . Notice, sketch for doc ends here. . . . . . . . . . . .


Documenting in greater details how I got Turtl client/agent working in same AppVM

Done by running local server and client in the same AppVM, without adding any kind of VM networking.

This... ripped a few teeth out on the way, but successful! Here's the step to reproduce as following;

1) First to keep original template clean; qvm-clone debian-9 debian-9-data-manager 2) Start the clone template, and install docker sudo dnf update && sudo apt-get install docker-ce 3) Add user to docker group, sudo usermod -aG docker user so non-root (sudo) can be avoided in use. 4) Shutdown template and create VM qvm-create turtl-builder -t debian-9-data-manager -l black, this is a temporary VM to be deleted later, also this AppVM must have internet access, for now. 5) In turtl-builder VM, run git clone https://github.com/ArthurGarnier/turtl-docker.git which contains docker files which build turtl server on a base ubuntu container build. 6) In turtl-builder VM, run cd turtl-docker and run sudo docker build -t turtl-agent . the dot at the end of command is important to not miss. It'll take some time, probably 5min on average from less to average CPU calculation machines. 7) Now to test the agent, run sudo docker run -d --name turtl -p 127.0.0.1:8181:8181 -v $(pwd)/volume:/var/lib/rethinkdb/instance1 -t turtl-agent and following up with confirming if it is running correctly with docker ps. It can be killed again with docker kill turtl-agent or instead use its ID if multiple of instances have the same -t attribute name. However keep it running, time to test the client!

Preparation towards the finish line 8) In turtl-builder VM, optionally create folder structure to keep it organized (however later commands take this structure into account, so it's recommended to use it). First cd ~ and then mkdir -p Turtl/{current-client-installation,data-server-backup,original-installer-files/{turtl-agent-docker,turtl-client}}

Client time! 9) Follow url https://turtlapp.com/ download & unpack the compressed files, but don't install yet. 10) To install, open turtl-builder VM, drag and drop the installer.sh into the terminal to generate the path to the installer.sh, and then afterwards write ~/Turtl/current-client-installation, which will put the client installation in this mobile transferable folder, thereby making the installation, and all the other sub-folders, mobile and transferable. Note the uninstaller can be done the same way, as the installer will inform at the end. 11) For a client desktop icon, write nano ~/Turtl/Turtl-client.desktop and put in the following 

[Desktop Entry]
Type=Application
Name=Turtl
GenericName=Secture notes
Comment=Private notes and bookmarks with collaboration.
Exec=/home/user/Turtl/current-client-installation/turtl
Path=/home/user/Turtl/current-client-installation
Icon=turtl.png
Terminal=false
Categories=Office;Utility;
Keywords=secure;security;privacy;private;notes;bookmarks;collaborate;research;
StartupNotify=true

close and save.

12) Double click the freshly made shortcut file, and click trust, which will open the client. 13) If the agent was shutdown earlier, then use sudo docker run -d turtl-agent to start it again. 14) In the started client, click advanced and then change the address to the local address. If the local address was forgotten, then it can be found with docker ps or alternatively use netstat or other networking tools. On my system its http://127.0.0.1:8181, presumably it'll be the same. 15) Now in the client, click create account, and explore the manager a bit.

Backups and transfer-ability 16) First shutdown the docker turtl-agent if it is running, check with docker ps . If it is indeed running, then docker kill turtl-agent, and confirm again if it's now shut down. 17) Now run docker save turtl-agent -o /home/user/Turtl/original-installer-files/turtl-agent-docker/turtl-agent-docker-backup.tar, this will not only save the server, but all the data within the server as well, all notes, pictures, bookmarks, etc. This command is what to be used to make on-going backups. 18) Now qvm-move offline-app-vm-name which either aldready exists or needs to be created. Note that docker must be installed in the template. If a new template is needed, then repeat step 1-3 up above. 19) Now in the offline AppVM, place folder in ~/home somewhere, and run docker load -i /path/to/latest/turtl-agent-docker-backup.tar to quickly bring back the docker turtl server, and all its data. 20) Optionally make/find a script that continuously run the docker save turtl-agent -o /home/user/Turtl/original-installer-files/turtl-agent-docker/turtl-agent-docker-backup.tar command to create backups, however, the script needs to be able to create new file-names rather than over-writing the previous file. Script must also prevent from running the command if the docker agent server is actively running, can be confirmed with docker ps, due to risk of corruption of a running operation system (it's kind of a mini OS after all).

Critical missing component

Currently docker keeps the loaded images in a non-persistent location of the AppVM, thereby when the VM is shutdown, the turtl server, and all changed data content, is lost. There are known steps to prevent this by changing which folders are persistent, and which aren't, but for now not included here.

Regarding memory usage

It seems it takes some RAM to run, some 500-600 MB in addition to the 400-500 MB default VM. It might get worse, I haven't tested this much yet.

one7two99 commented 6 years ago

A friend told me that it is not possible to store formatted text and pictures in turtle?

This something which works well with OneNote or evernote.

Can someone comment on this?

[799]

Aekez commented 6 years ago

@one7two99 You can definitely store pictures and format text in Turtl, not sure how he made that conclusion? I'll try see if I can make some demo's illustrating how Turtl looks like.

The formatting is actually very much the same type of formatting you can do here in docs, with ### ~~ markdown format, perhaps this is what he means by no formatting? it's there, just more hidden. Turtl doesn't have buttons you can click on, it's acting the same way as GitHub formatting in the markdown pages, except unlike GitHub it has a panel which gives the users insight into the formatting rather than expecting users to know all formatting tricks in advance. I'm not sure if all formatting tricks are listed there or not, but at least it has all the basics.

Despite Turtl being the best Note Manager I know of, I haven't used Turtl much my self yet due to the remote public server issue that was a very big turn-off for me, but now that I got into building the server, Turtl definitely seems redeemed to me, as this was the one and only, but major reason that I never started using Turtl. Perhaps I'll be disappointed as well once I use it more, time will tell, I need to try use it for a longer period of time my self. But Turtl is still currently the best I know of though, with CherryTree Note-Manager coming in close second (not in terms of features, otherwise OneNote/Evernote would probably win, but in overall holistic view, open source, owning your own data, if its actively maintained and improved, etc.).

btw I recommend trying out a testing account on the public Turtl server, here it's really fast to test this out. It's the making of your own server that takes all the effort, like in this article, so if you skip the private server, then you can quickly test it on the public server if it's for you or you prefer something else. If you like it, then you can always build your own server afterwards :)

Also you can use any name or password, and you don't need to provide your e-mail (it'll ask for e-mail, but you can easily skip it as it's only optional).

I might have some time today to finish the offline AppVM concept (still need to find best choice between changing AppVM persistence, or if I can get docker to save its settings in /rw, then I'll give it a shot by testing Turtl itself afterwards.

Hopefully we can get a more clear picture soon of the remaining uncertainties :)

Aekez commented 6 years ago

@one7two99 Made this quick-guide to help saving time to testing out Turtl before investing too much time in it :)

Quick speed steps to testing out Trutl on public server while running Qubes 1) download linux client https://turtlapp.com/ 2) Unpack and write or drop the installer.sh from the download into the terminal to automatically write the address, and then space key and add the extra line /home/user/Turtl-client/

This will install Turtl client in your home folder, and preserve Turtl between re-boots. It's also easy to uninstall, do the same as above, but include "uninstall" afterwards. But most of the install is in the folder, it'll only create some shortcuts in ./local/share/applications and so on. So the uninstall doesn't do much anyway, since it's easy to manual delete.

3) Run the client in an AppVM with internet access in AppVM terminal /home/user/Turtl-client/turtl. 4) Create testing account. You do not need to type in e-mail or anything else but name and password. E-mail is only if you forget your login at some point, but it can be skipped too.

5) You can optionally also test the smartphone Turtl app since this testing account is a remote internet server. Of course it can also be used permanently if you feel ok trusting the Turtl hosting provider :)

Aekez commented 6 years ago

To minimize clutter, this guide/doc will be copied to QCC/content/ section, where update suggestions also may be submitted more freely, while discussions for this can be continued in the parent topic, clean browsing at https://github.com/Qubes-Community/Contents/issues/24

Above content may eventually be deleted in order to avoid confusion between new and old versions.

one7two99 commented 6 years ago

Hello,

On 05/12 08:11, Yuraeitha wrote:

@one7two99 You can definitely store pictures and format text in Turtl, not sure how he made that conclusion? I'll try see if I can make some demo's illustrating how Turtl looks like. The formatting is actually very much the same type of formatting you can do here in docs [...]

I have taken a look at turtl but it seems that working with pictures AND text is a bit more complicated compared to evernote, but I will make some more investigation and will setup an own turtl server. As having notes synchronized between devices and I will not store very confidential data there I think about setting it up on a raspberry pi to which I can connect via VPN.

One missing feature is that I can't save a webpage or parts to it, like I do with Evernote. Example: finding a nice recipe -> save to evernote.

but maybe we see improvements here in the future.

[799]

Aekez commented 6 years ago

It's a cool idea to use the raspberry pi for non-critical data, maybe we could include a separate segmentation instructions on how to do this. It definitely solves the memory issue of RAM though.

I haven't used Evernote since forever, so I can't easily relate to how it works now a days. But hopefully it isn't too annoying or a too big of a downgrade for users to go to Turtl from Evernote. But I definitely do not want to keep this project alive artificially either. It needs to bring value in some way to the user to make it worth it. A particular big worry I have about keeping the offline server in the same AppVM as the client, is the additional RAM Turtl ends up using with the server. It's afterall essentially an operation-system within an operation-system, within an operation-system (Xen/Qubes --> Template/AppVM's --> Docker/Turtl-server --> Turtl client). I really hope this one can be mitigated a bit, at least to some extent. I still need to give it a try to see if I can get it to behave better on RAM use. Like you, I don't feel truly safe with remote servers, not even if in private control. So solving this ram use is important, unless we can find a better Turtl replacement. But then again, if this can work, then it would be a nice to feature to have the ability to move Turtl notes between an online and offline version of it. How feasible all this is is still a bit uncertain, but lets try see if we can make it work without too great limitations / sacrifices to ease of use, productivity or system resources.

About moving content into Turtl from a browser, I think its possible with the firefox Turtl addon. I noticed the firefox addon link is dead on the Turtl site download section (bit worrisome that it wasn't fixed), so I went looking in firefox database, and it seems the link must have changed cause its still there (or maybe the addon changed) https://addons.mozilla.org/en-US/firefox/addon/turtl-bookmarking/?src=search I haven't tested the addon my self yet though, I'm not sure how well it works. But it looks very interesting. Maybe it's possible to modify the addon to send clear text html/css to the offline AppVM with the offline Turtl? I'm not sure how feasible this would be though.

I apologize for being slow, I also have to cram for soon up-coming exams, so it becomes a bit problematic when issues take longer than expected. I ended up with issues on how to handle the AppVM persistence in relation to docker. I wanted to make everything mobile and easily transferable from one AppVM to the next AppVM, and there are some alternate routes to take, which involves various different problems. For example making docker directory persistent means it takes more work to keep it updated. But making it non-persistent, then it'll be updated along the template updates, however, any active server data will not be copied along the mobile folder if not first making a backup of the docker image to be included in the mobile folder transfer (when moving to new AppVM's), and in addition it does not preserve the docker image/changes between AppVM restarts. The main issue being docker doesn't seem very modular, the docker image and docker itself, seems too chaotically mixed together.

The problem then boils down to, whether it is possible in one way or another, to split up docker itself, and docker image data, so that only the docker image data is on the persistent AppVM folders, while keeping docker app itself in the non-persistent folders. This also makes it easy to update docker, or being able to move the AppVM containing the docker image to a fresh template if no direct docker backup was done, but one still has a newer Qubes AppVM backup. It's problematic how much of a mess docker seems to be when it comes to being modular, it's nothing like Qubes in that regard. Perhaps I misunderstood how docker works, but so far, it just seems like a giant interconnected system. I'll need more time to figure out how docker really works though, hopefully I can get some time for it soon, and whether it is possible to split the two from each others.

Re-opening issue for active Turtl specific discussions.

awokd commented 4 years ago

Closing for now.