Publish Issues

[Aetherinox]

Hey, had to step away, this is part 2 of #24

In short, the issue still exists that I cannot view the online published MD files using publish.domain.com (with your nginx template). The site itself works and I get json formatted error messages, but it can never find the pages / slug.

So I tried out a series of tests. When originally executing:

curl --request POST --url https://api.domain.com/publish/list --header 'Content-Type: application/json' --data '{ "token": "xxx"}'

It would return:

{"limit":1,"shared":[],"sites":[]}

I tried numerous re-creations of the vault, wiped info, etc. Nothing ever worked.

I then manually entered:

curl --request POST --url https://api.domain.com/publish/create --header 'Content-Type: application/json' --data '{ "token": "xxx"}'

And it created a slug for me as 755625b4-86c5-43cd-84fa-89f2b415596c

I used that slug as part of the domain url https://publish.domain.com/SLUG_HERE and I actually get information returned in json now

{
  "error": "record not found",
  "path": "",
  "site": {
    "id": "3540989e-c7c6-5aac-ba5f-7e7127baac43",
    "host": "domain.com",
    "slug": "755625b4-86c5-43cd-84fa-89f2b415596c"
  }
}

Before I was just getting Record Not Found and nothing else.

I'm not sure if I'm onto something here, I'm just tracing all the code backward to figure out where the issue could be, in case any of this helps out. It seems like my vaults aren't getting a slug, so the webserver can't find the pages. I've tried numerous vaults now, and can access no .md pages.

Edit: Did a wipe and started fresh. Linked obsidian account, enabled sync / public and then synced my starter vault. Which gives me the sites list of:

{"limit":1,"shared":[],"sites":[]}

Decided to use this command to see if I could assign a slug to the newly created vault uid with:

curl --request POST --url https://obsidian.domain.com/api/slug --header 'Content-Type: application/json' --data '{ "id": "06d68834-b7e4-42e9-b72a-592d456ff3a3", "slug": "Test", "token": "xxxx"}'

Which returns

{"error":"You do not have permission to change this site's slug"}

Attempted to access the online published site using the URLs:

https://publish.domain.com/Test/Welcome.md
https://publish/domain.com/06d68834-b7e4-42e9-b72a-592d456ff3a3/Welcome.md

Test being the name I made for the vault, and the 2nd one being the UID auto assigned to the vault to see if either one of the two would work.

Both URL attempts return

{"error":"Site not found"}

On a positive note, after the endless hours of screwing around with this, I was able to edit the docker-compose file and get the vault database to save in a specified location. For some reason, the default docker-compose template doesn't have a period at the front of the path, but as soon as I added a period, it started saving in the folder where I originally specified it to go.

[acheong08]

And it created a slug for me as 755625b4-86c5-43cd-84fa-89f2b415596c

Looks like you made a mistake by trying to work through it via curl. The publish plugin makes 2 requests to create a site, one to create and another to set the site (slug) name

[acheong08]

Screencast from 2023-09-09 15-42-41.webm

[acheong08]

The flow goes something like this:

1. Check if the site already exists:

   
   fetch("https://api.domain.com/api/site", {
   "body": "{\"slug\":\"generic\",\"token\":\"<token>\"}",
   "method": "POST"
   });

{"code":"NOTFOUND","message":"Slug not found"}

2. Create a new site
```js
fetch("https://obsidian.duti.me/publish/create", {
  "body": "{\"token\":\"<token>\"}",
  "method": "POST"
});

{
    "id": "514e3e1a-f5d7-46d2-83d0-0c6fa7b5a584",
    "host": "api.domain.com",
    "created": 1694245459098,
    "owner": "<your email>",
    "slug": "a0ccf713-7b3c-4b82-918d-d3824c4b1e42",
    "size": 0
}

Ignore the slug.

4. Set the actual slug

   fetch("https://obsidian.duti.me/api/slug", {
   "headers": {
   "accept": "*/*",
   "accept-language": "en-US",
   "content-type": "application/json; charset=utf-8",
   "sec-ch-ua-mobile": "?0",
   "sec-ch-ua-platform": "\"Linux\"",
   "sec-fetch-mode": "cors",
   "sec-fetch-site": "cross-site"
   },
   "referrerPolicy": "strict-origin-when-cross-origin",
   "body": "{\"id\":\"514e3e1a-f5d7-46d2-83d0-0c6fa7b5a584\",\"host\":\"api.domain.com\",\"slug\":\"<actual slug>\",\"token\":\"<token>\"}",
   "method": "POST"
   });

[Aetherinox]

Just woke up a bit ago, so my brain isn't fully functioning yet.

Which interface are you on where it says "Manage Sites". Because I've been to the Sync section, and the one setting for the plugin itself, and I can't seem to find this.

The places I've been modifying settings from was Sync -> Remote Vault

I don't see an interface at all such as yours. Unless I'm missing a critical part.

And if I go to "Core Plugins" and locate "Publish", I see no settings button at all.

[acheong08]

[acheong08]

[acheong08]

I don't see an interface at all such as yours. Unless I'm missing a critical part.

I think now I understand the confusion.

[Aetherinox]

Yeah I've been ALL over this program today trying to see how a slug is created, and I completely skipped that part. I have that button, but mine throws a network issue.

[acheong08]

Yeah I've been ALL over this program today trying to see how a slug is created, and I completely skipped that part. I have that button, but mine throws a network issue.

Check the network log. Have you updated the plugin?

[Aetherinox]

I double checked the Sync log to ensure it pushed all my files, and it connected fine. Also made sure I'm signed in via Obsidian. Are you referring to a special network log feature in Obsidian, or developer window.

[acheong08]

developer window.

yes. ctrl+shift+i should open it

[acheong08]

The publish feature is completely separate from sync

[Aetherinox]

Yeah, starting to see that. Should have seen it this morning, but I was digging into something much more complicated. Part of the issue is I have too many plugins including the left bar, so I would have never noticed a new icon.

In regards to network, the only entry I'm seeing is:

The URL seems to be different. It's looking for https://domain.com/api/list whereas, my subdomain is obsidian.domain.com, does the URL have to be API as the subdomain?

[acheong08]

The URL seems to be different. It's looking for https://domain.com/api/list whereas, my subdomain is obsidian.domain.com, does the URL have to be API as the subdomain?

No. Did you perhaps set the DOMAIN_NAME environment variable to something other than your subdomain?

[acheong08]

obsidian.domain.com

That is how my domain name looks as well

[Aetherinox]

Originally I had:

DOMAIN_NAME=domain.com

I now changed it to

DOMAIN_NAME=obsidian.domain.com

Restarted docker container, and it's still showing the original request url of domain.com/api/list, unless I'm missing something really damn simple here.

And obsidian.domain.com is a valid subdomain, working properly from browser.

[acheong08]

Restarted docker container, and it's still showing the original request url of domain.com/api/list, unless I'm missing something really damn simple here.

The domain name is stored in the database. This was to allow horizontal scaling (incomplete). You need to delete publish.db and restart the container

[Aetherinox]

Done, gave obsidian a reboot just in case

Now it's showing slugs as one successful and one fail

Same domain as before with publish.db wiped and docker container restarted

[acheong08]

I'm not quite sure why it would still use the old domain name. Try clearing all the databases & deleting /path/to/your/vault/.obsidian/publish.json

[Aetherinox]

Yeah, doing all that now. Even wiping the docker container just to make sure something isn't caught.

[Aetherinox]

I swear this is so bizarre. Sync has always worked, but this time around after creating a new user, the log has:

2023-09-09 01:32 - Connecting to server
2023-09-09 01:32 - Did not respond to login request: {"error":"record not found"}

2023-09-09 01:32 - Connecting to server
2023-09-09 01:32 - Did not respond to login request: {"error":"record not found"}

2023-09-09 01:32 - Connecting to server
2023-09-09 01:32 - Did not respond to login request: {"error":"record not found"}

2023-09-09 01:33 - Connecting to server
2023-09-09 01:33 - Did not respond to login request: {"error":"record not found"}

2023-09-09 01:33 - Waiting to connect to server

Only change made was changing the env from domain.com to obsidian.domain.com

But now, Publish works, and I can create.

[acheong08]

I swear this is so bizarre. Sync has always worked, but this time around after creating a new user, the log has:

Deleting the databases means you need to sign up & log in again.

But now, Publish works, and I can create.

The reason publish works but sync doesn't is because the server uses JWT for tokens, meaning that a deleted user can still use certain functions for some time until the token expires. Might need to change that

[Aetherinox]

I swear this is so bizarre. Sync has always worked, but this time around after creating a new user, the log has:

Deleting the databases means you need to sign up & log in again

Yeah, did all that right after the container was restarted. I feel like I could start this damn thing blind-folded now. I don't know why, but Obsidian wanted me to disconnect from the sync and re-connect, and it started pushing after that. That didn't happen before.

On the shocking side of things, publish works, it uploaded my MD files

2023-09-09 01:37 - Uploading Windows/PowerShell/BurntToast
2023-09-09 01:37 - Uploading Windows/Environment Variables

And I can now actually view the .md file code. Aside from the fact that I feel like I can't even breathe on it and pray to the tech gods that nothing dies. It appears to be working.

Sorry this went in a direction so far outside the realm of normal. Not seeing that publish button royally screwed me up. I could have probably diagnosed it had I seen that. Because once I would have gotten fed up enough of it not connecting, I would have wiped again anyway. I've done it about 10 times today.

I feel like I need to create a damn Obsidian note on this plugin alone just of all the DOs and DONTs so if I need to re-configure it, I can remember everything.

Unfortunately, I do C#, PHP, Lua, and JS. I've messed with Go for about 5 minutes of my life, and I still don't have a full grasp of how Obsidian and plugins communicate together. Nor do I have the head room right now to learn it.

All I do know is this plugin is gold. I like to self-host everything. I get that Obsidian is secure by their standards, but I feel much more comfortable with my data being on my machine with me in control of security. Out of 80 plugins my vault has, this is easily one of my top 3.

Appreciate the hand holding, and sorry again this went WAY off the rails.

[acheong08]

Nice that it went well. Sorry about the trouble it gave you. I'm horrendous at documentation (hopefully will improve)

[Aetherinox]

Nice that it went well. Sorry about the trouble it gave you. I'm horrendous at documentation (hopefully will improve)

No worries. I'm going to write up my own doc on it just so I've got troubleshooting once I get a firm understanding of how it functions. I'm going to at least write down the flow of commands you provided above so I know the basics of how it does its thing.

Appreciate the help again.

[acheong08]

No worries. I'm going to write up my own doc on it just so I've got troubleshooting once I get a firm understanding of how it functions. I'm going to at least write down the flow of commands you provided above so I know the basics of how it does its thing.

Could you perhaps share your docs when you complete them? As someone who has no idea what docs are supposed to look like, it would be a great reference for improving the readme/wiki. Might also help out anyone that faces the same issue. Thanks

[Aetherinox]

Could you perhaps share your docs when you complete them? As someone who has no idea what docs are supposed to look like, it would be a great reference for improving the readme/wiki. Might also help out anyone that faces the same issue. Thanks

Sure thing. I'll send you the link once I get them written up. Being in development over 15 years (and as you've seen in the last 24 hours with me), sometimes what may be painfully obvious to the developer, ends up being just a pain for the user. I've had to create docs that pretty much hold the person's hand because the better the docs, the less you get annoyed by tickets and the more time you get to work on a product.

I'll just drop them in a repo with a readme and link you when I'm done. Got to create screenshots and all that jazz.

[Aetherinox]

@acheong08 Definitely not done, but I put the overall template together and I need to change some things from my configuration to the default and add sections for configuring the plugin, etc, but this is the just of my doc: https://github.com/Aetherinox/obi-sync-docs

Might be good as a readme, but even more details within the repo's wiki would be awesome.