Add instructions to setup your own CouchDB to work with NoteSelf

[Mirodin]

I really like the way NoteSelf handles many wikis in one file. I just wrote in the GG about my problem with a lot of wikis on a Raspberry with limited memory and that it gets stuck after so many Node instances (https://groups.google.com/forum/#!topic/tiddlywiki/xWpNnABLYbQ). As I do not feel comfortable uploading my data to Cloudant (or any other third party) I tried to setup my own CouchDB. However this does not work for me, I am not able to login when NoteSelf prompts me to, it always returns a XMLHTTPError - even when I try to login as an admin. So maybe you could drop a few lines about what to look out for if one decides to setup a local instance.

Thank you anyways for this great peace of software :)

[thejeshgn]

@Mirodin Are you trying to set CouchDB on home server or on a cloud server?

[Mirodin]

I am trying to use a CouchDB on my home server. CouchDB self-check runs fine and I can access it via my browser so there should be no problem on that side. But I can not manage to make NoteSelf talk to the database even though I enabled CORS from all domains. I think I miss a crucial part when it comes to my setup.

[georgeogoodman]

...and I would like to install couchdb on a remote server (I have one hosted on A2Hosting) for use with NoteSelf. CouchDB is installed and I'm just starting the "how do I configure this" journey.

Any wise words you can add to the NoteSelf docs for this model would be a great gift (in addition to the gift you've already given with NoteSelf!).

[danielo515]

Hello @georgeogoodman and @Mirodin Thank you for giving NoteSelf a try. I agree with you that some instructions about setting up your own server should be added to the main page. Probably creating a new section about instructions will be a good idea. For now, I think it can fit into the FAQ. Someone wrote a comprehensive tutorial about how to setup this on your own server. I am not sure if it was on my email account or if it is buried on google groups.

I'll look for it. If you succeed in executing this instructions I would love to see a PR adding such instructions.

Regards

[willwray]

As you may know, the cloudant.com Shared Plan is being retired on March 31st Shared Plan Retirement FAQ

Help...

I've been using NoteSelf synced to cloudant.com db, as set up based on danielo's video instructions. In preparation for March 31st, I signed up with IBM bluemix, created a CouchDB instance and replicated my data - the wiki files plus the _replicator and _users databases.

Now, I can't get NoteSelf to login to my new bluemix DB: PouchAdaptor - ... (date/time / count) ... Login failed I've added credentials and API keys on bluemix and tried those, as well as my old cloudant login.

I guess that a change is needed to the _users db file, to work with the new db and credentials.

Any hints? Or any more info needed to help diagnose?

I really appreciate NoteSelf with its remote db sync, I'd really like to keep using it. Thanks

[creinig]

I have the same problem as @willwray - noteself always fails to login with IBM bluemix cloudant as well as with a cloudant DB hosted on www.smileupps.com. Access to the couchdb instance from my browser works perfectly fine -- both from the firefox I use noteself in & administer the couchdb instances and from a chrome that doesn't have any credentials stored.

Some research came up with info that the old cloudant couchdb hosting used couchdb 1.0, while at least smileupps.com is at 1.6 -- maybe that causes some problems.

Side note: Until this morning I assumed that noteself development was on ice, since I saw no activity in any of the github repos nor on the webpage. Only Your post https://groups.google.com/d/msg/tiddlywiki/9JH1V0Ms4zs/LwGVpy2lDQAJ made me realize that the noteself copy downloaded from the website actually differs from the one I'm using so far :) Still doesn't work though...

Many thanks for noteself -- it's really an amazing help for me and I'd hate to lose it :)

[danielo515]

NoteSelf development is not on ice. It's just that it is good enough for me, which happens to be the only developer and the only source of innovation, so as long as I don't need anything new, I will not spend time developing it. That said, NoteSelf is not going anywhere because I also use it on a daily basis. I'll try to perform the migration myself to see if the same problems happens to me.

Regards

[danielo515]

I just created a new account on IBM bluemix (which was a tedious, hard and confusing process) and I have just deployed a cloudant instance. After that I just went to NoteSelf online, opened an existing "notebook" (AKA database) and just pasted the new url under the CouchDB URL: field. This field is on the control panel, tiddlypouch tab and then database. The url is now much uglier, but it is on the form of https://random-and-long-string-with-numbers-bluemix.cloudant.com Don't try to use the replication mechanism of cloudant, noteself will take care of the sync process to the new location, and everything should work out of the box.

Please make sure you try from the online version: https://noteself.github.io/online/ Let me know how well that goes. Regards

[creinig]

Thanks a lot for the fast response. It works now. The culprit was the disabled CORS in the bluemix couchdb. Once I enabled that everything worked just fine. Sorry for the hiccup.

And yes, the IBM cloud is a very unpleasant experience. Which is why I'm also trying out the smileupps option.

[creinig]

Update: the smileupps.com couchdb works as well. CORS can be enabled via Fauxton (activate via the button next to the DB in the config panel). It's marked as "experimental", but the also provided Futon seems to have problems with editing the config. http://docs.couchdb.org/en/1.3.0/cors.html describes the config settings to modify.

[danielo515]

Glad it is working for all of you. @willwray maybe you can enclose the technical instructions between triple ticks so they are properly rendered

[willwray]

Thanks Daniel and creinig,

I have it working on my bluemix cloudant now. The cause? It was CORS. Of course! I couldn't work out how to configure CORS from the bluemix interface so I followed the IBM docs using the Python API:

https://console.bluemix.net/docs/services/Cloudant/getting-started.html#getting-started-with-cloudant https://console.bluemix.net/docs/services/Cloudant/api/cors.html#cors

On my linux box:

$ sudo pip install cloudant
Downloading cloudant-2.8.1-py2-none-any.whl
$ ipython
IPython 6.2.1 -- An enhanced Interactive Python. Type '?' for help.

Python code (ipython session)

In [1]: from cloudant.client import Cloudant
In [2]: from cloudant.error import CloudantClientException
In [3]: from cloudant.result import Result, ResultByKey
In [4]: client = Cloudant( "my username", "my password", url="username:password@bluemix-url")
In [5]: client.connect()
In [6]: client.update_cors_configuration(origins=["https://noteself.github.io"])
Out[6]: {u'ok': True}

Thanks Daniel for this really useful noteself setup and for your quick support.

[diego898]

@danielo515 (or anyone :] ) did you find the instructions for setting it up locally? thanks!

[danielo515]

If anyone feels brave and want to write a manual the brand new official forum is a good place for it: https://forum.noteself.org

[johnd0e]

If anyone feels brave and want to write a manual

Well, I could try, but in current state cloudant is not actually usable option, as long as it requires you to relogin 100 times a day (and relogin means not just reenter credentials, no, you have open real cloudant dashboard, and proceed through several redirections).

I saw api key field in NoteSelf ui, but failed to find out what it should be (I tried to generate several different api keys, but none of them worked).

Perhaps, there were some changes in login process on ibm side, and NoteSelf is not updated yet with that changes?

[godestalbin]

I have the same behaviour as @johnd0e: I need to reconnect to cloudant dashboard several times a day in order to keep Noteself able to synchronize with my cloudant couchbase DB. When Noteself shows it is not more synchronizing, when I click the Offline button, a popup login/password windows pops up but my credentials cannot be validated.

[danielo515]

@johnd0e @godestalbin I really encourage you to ask this kind of questions on the forum. However, I will answer your problem directly here. NoteSelf works perfectly fine with the new version and plans of IBM cloud. However, the process of getting the credentials and the information has changed, and probably you are confused by this. I'll try to record a video about how to properly get the credentials.

In any case, you are invited to try the free sync mechanism that NoteSelf offerts. It's only limitation is that you can only have one database

[danielo515]

Here is the video I promised, hope it helps you

https://youtu.be/Ix55pZ-WTvI

[melvinroest]

Alright, I'll write a quick 'n dirty tutorial on how to host Noteself on your own computer. I'm assuming a linux operating system. I recommend to have some old computer that can run 24/7, because that's what you'll do :D

Limitations: I can't get it to work on my phone for some reason, but it did sync on multiple laptops and browsers.

Another limitation: I'm not doing things cleanly, I'm just doing it so I can get it to work and I'm (kind of) fine with being unsafe.

Following this guide is at your own risk.

What to do:

0. Install Ubuntu (why Ubuntu? You wouldn't read this guide if you were a more hardcore Linux user ;-) Plus Ubuntu is the most user friendly Linux). Check some online guide on how to do it. Raspberry Pi users, obviously have the knowhow to deviate from this. I use an old Mac Mini, so Ubuntu it is.
1. Put your computer open to the internet by forwarding your ports.
2. Install Couchdb either via Bitnami (if on 19.04) or otherwise via their installation guide.
3. Disable CORS
4. Host your own noteself file on your computer.
5. Serve it up with Python.

1. Put your compute open to the internet

Part A: get your internal IP-address

Some theory:

You're at home. You have a router (the thing that you get internet from). From your internet provider you get a static IP-address, which means that the IP-address stays the same. If you don't have this, then this guide is not for you.

Your router does this thing called "Network Address Translation", because your router assigns different IP-addresses to your internal network starting with 192.168.0.0 to 192.168.255.255. So if your actual IP is 111.111.111.111, your router will change that to, for example: 192.168.1.1

You need to know the internal IP-address of your computer. Type ifconfig -a in terminal. You get data like:

wlp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.178.14  netmask 255.255.255.0  broadcast 192.168.178.255
        inet6 fe80::9128:89cf:22d9:e18d  prefixlen 64  scopeid 0x20<link>
        ether a8:86:dd:a6:f2:9f  txqueuelen 1000  (Ethernet)
        RX packets 387544  bytes 436562304 (436.5 MB)
        RX errors 0  dropped 0  overruns 0  frame 93958
        TX packets 264914  bytes 152233935 (152.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17

My internal IP here is 192.168.178.14.

Part B: forward your ports.

Some theory:

Your router determines network address translation. Aka it is the gatekeeper for everything. It allows you to make a connection to another computer, but another computer cannot just easily make a connection with you. All the ports are closed. So you need to open a port up.

Here's how to do that:

1. Log in to your router (I walked to my router and read what IP-address I should type in and what username and password was).

2. Forward your ports: couchdb by default uses port 5984. So you need to state something like the following: Port to open up: 5984 Forward to port: 5984 (this is the port your router will expose, it can also be another port, but you can also keep it the same) From internal IP-address: 192.168.x.x

You will also need to do this for another port for your Python server. I used 8888.

Done

2. Install Couchdb

I downloaded the bitnami stack, made it executable with chmod +x the_file_name_of_the_downloaded_file and then ran it with ./the_file_name_of_the_downloaded_file

You get a visual installer and click next.

3. Disable CORS CORS simply stands for: different websites cannot simply communicate with each other via JSON in order to invoke actions on the other server. It's a lot more than that, and what I say is slightly wrong, but that's the gist of it.

I edited the local ini file.

Open it up via nano your_installation_dir/couchdb/etc/local.ini.

Put this at the end

[cors]
origins = *
credentials = true
methods = GET, PUT, POST, HEAD, DELETE
headers = accept, authorization, content-type, origin, referer, x-csrf-token

Also put bind_address=0.0.0.0 below [chttpd] and [httpd].

4. Host your own noteself file on your computer. I'm assuming you have physical access to the computer.

Put your noteself.html file on there. Rename it to index.html and put it in a folder.

5. Serve it up with Python.

Go to that folder with your terminal and serve it up with Python by typing python -m SimpleHTTPServer 8888 (or any other port number).

Test it by typing in your IP-address and your port number.

You can also checkout your couchdb instance at http://YOUR_IP_ADDRESS:5984/_utils/ and create users and databases.

The standard username is: admin.

Edit: I think it's also the case that one needs to have no admin credentials.

So I did the following:

1. clear the [admin] section in your install_dir/etc/couchdb/local.ini, also set require_valid_user=false (it is probably on require_valid_user=false).
2. delete the _users.couch database (located in install_dir/var/lib/couchdb/)

Note: all of this is very unsafe, but there are a couple of ways to improve it (e.g. cors only on one domain. Create a simple authentication website in front of it, etc.)

[danielo515]

Thanks for the writeup @melvinroest, and thanks for the warnings. My intention is to record a new video about how to do this safely on Amazon cloud for very few money, but it is not on my priority list, I prefer to focus on my easy to opt in official sync service