Publishing with quartz sync and symlinks fails

[reallyely]

Describe the bug

With a quartz app set up with it's content hosted as a symlink, publishing to Github Pages results in an empty app.

To Reproduce Steps to reproduce the behavior:

1. Create a new Empty Quartz app
2. Create content folder with symlink New-Item -Path content -ItemType SymbolicLink -Value C:\path\to\vault\<vault>
3. run npx quartz sync

C:\dev\quartz [main ↑1 +1 ~0 -0 !]> New-Item -Path content -ItemType SymbolicLink -Value C:\vault 
C:\dev\quartz [main ↑1 +1 ~0 -0 !]> npx quartz sync --no-pull

 Quartz v4.2.3

Backing up your content
Detected symlink, trying to dereference before committing
warning: adding embedded git repository: content
hint: You've added another git repository inside your current repository.
hint: Clones of the outer repository will not contain the contents of
hint: the embedded repository and will not know how to obtain it.
hint: If you meant to add a submodule, use:
hint:
hint:   git submodule add <url> content
hint:
hint: If you added this path by mistake, you can remove it from the
hint: index with:
hint:
hint:   git rm --cached content
hint:
hint: See "git help submodule" for more information.
hint: Disable this message with "git config advice.addEmbeddedRepo false"
[main a6ba7fc] Quartz sync: Jul 12, 2024, 10:05 AM
 1 file changed, 1 insertion(+)
 create mode 160000 content
Pushing your changes
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 16 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 411 bytes | 411.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/really-ely/notes.git
   bc28eb5..a6ba7fc  main -> main
branch 'main' set up to track 'origin/main'.
Done!

Git remote updates Actions run successfully Pages URL is published

Expected behavior

• Content Folder on remote to have content
• GitHub Pages URL generated from Actions to navigate to index

Actual

• Content folder resembles submodule in remote
• visiting published URL displays index.xml. Other pages return 404
• Local symlink folder displays as empty but admin user no longer has permissions to read it

Desktop (please complete the following information):

• Quartz Version: e.g. v4.2.3
• node Version: v18.19.0
• npm version: v10.7.0
• OS: Windows 11
• Browser Brave

Additional context

• Powershell 5.1
• deploy.yml copied from documentation
• branch changed from v4 to main
• The Obsidian vault that I was trying to symlink to from Quartz was a OneDrive folder

[reallyely]

• I manually copied my vault over to the content folder to see if I could get it to deploy in a more direct use case. It was successful.
• I then locally deleted the quartz/content folder
• I moved my Obsidian vault out of my one drive folder, created a new sym link and re-ran npx quartz sync.
• The remote still had a static content folder
  • it's contents showed last updated at the time of my last push
  • Updated contents from the symlinked folder now also showed up
  • The website rendered as expected with updated content

However, the content folder in my quartz directory is still unreachable after the sync

C:\dev\quartz [main ≡ +1 ~0 -61 !]> ls content

    Directory: C:\dev\quartz

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---l         7/12/2024  11:02 AM              0 content

C:\dev\quartz [main ≡ +1 ~0 -61 !]> cd content

cd : Cannot find path 'content' because it does not exist.
At line:1 char:1
+ cd content
+ ~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (content:String) [Set-Location], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.SetLocationCommand

[reallyely]

I thought I'd share my solution to this problem that directly serves my needs. It's opinionated but may help someone else.

I chose to skip using quartz sync and write my own solution. I'm still using the deploy.yml that is in the getting started guide, but I created a specific branch for publishing called publish. The general strategy is to locally check out a branch to copy the symlinked folder contents in, pushing that and letting the build run. I then switch back to my original branch and delete the local publish. I force push to publish with the assumption that I always want the latest state and don't want to resolve conflicts.

Additionally, I wanted it to continuously watch for changes and publish at regular intervals to keep things immediately up to date. I'm a noob at this file management stuff so my hashing attempt may be naive and costly but it's working for my small garden.

I like that this keeps main focused on the application, and publish on the contents and we don't need to worry about restoring the symlinked folder.

For context, I have had an Obsidian vault for many years now and wanted a permanent solution for sharing those core notes with separate vaults I create for individual jobs. Usually those job-related ones need to be locked down in some way and live and die on their infrastructure so I symlink my core notes into the job.

With this script I'm able to work seamlessly between both vaults and publish all of that to the web continuously.

[reallyely]

import assert from "assert"
import crypto from "crypto"
import { execSync } from "child_process"
import fs from "fs-extra"
import { hideBin } from "yargs/helpers"
import ora from "ora"
import path from "path"
import process from "process"
import yargs from "yargs"

interface Arguments {
  interval: number
}
const argv = yargs(hideBin(process.argv))
  .option("interval", {
    alias: "i",
    description: "Interval in minutes between checks",
    type: "number",
    default: 5,
  })
  .help()
  .parse() as Arguments

const checkIntervalMinutes = argv.interval as number
const checkIntervalMs = checkIntervalMinutes * 60 * 1000
const contentPath: string = "./content"
const publishBranch: string = "publish"
// State used to detect when we need to update
let previousHash = ""
let lastCheckTime = new Date()
let lastPublishTime: Date | null = null
let publishStatus = "Not published yet"

const spinner = ora({
  text: "Initializing...",
  color: "yellow",
  spinner: "dwarfFortress",
}).start()

function updateSpinner(text: string, color: "yellow" | "red" = "yellow") {
  spinner.color = color
  spinner.text = text
}
function persistMessage(message: string, error = false) {
  const timestamp = new Date().toLocaleTimeString()
  spinner.stopAndPersist({
    symbol: error ? "❌" : "✔",
    text: `[${timestamp}] ${message}`,
  })
  spinner.start()
}

function execCommand(command: string): string {
  return execSync(command, { encoding: "utf8", stdio: "pipe" }).trim()
}

async function main(): Promise<void> {
  const originalBranch = currentBranch()

  // Step 4: Switch to publish branch
  updateSpinner("Switching to publish branch")
  execCommand(`git checkout -B ${publishBranch}`)

  // Step 1: Check if content is a symlink
  updateSpinner("Checking content folder")
  const contentStats = await fs.lstat(contentPath)
  if (!contentStats.isSymbolicLink()) {
    throw new Error("content is not a symbolic link")
  }

  // Step 2: Get the target of the symlink
  const linkTarget = await fs.readlink(contentPath)

  // Step 3: Create a perfect copy
  updateSpinner("Creating copy of content")
  const tempPath = `${contentPath}_temp`
  await fs.copy(linkTarget, tempPath, {
    dereference: true,
    preserveTimestamps: true,
  })

  // Step 5: Replace symlink with actual folder
  updateSpinner("Replacing symlink with actual folder")
  await fs.remove(contentPath)
  await fs.move(tempPath, contentPath)

  // Step 6: Commit changes
  updateSpinner("Committing changes")
  const date = new Date().toISOString().split("T")[0]
  const time = new Date().toTimeString().split(" ")[0]
  execCommand(`git add ${contentPath}`)
  execCommand(`git commit -m "Content update: ${date} ${time}"`)

  // Step 7: Push changes to publish branch
  updateSpinner("Pushing changes to publish branch")
  execCommand(`git push -f origin ${publishBranch}`)

  // Step 8: Switch back to original branch
  updateSpinner("Switching back to original branch")
  execCommand(`git checkout ${originalBranch}`)

  assert.strictEqual(
    currentBranch(),
    originalBranch,
    `Failed to return to original branch. Expected ${originalBranch}, but on ${currentBranch}`,
  )

  updateSpinner(`Deleting ${publishBranch} branch`)
  execCommand(`git branch -D ${publishBranch}`)

  // // Step 9: Restore symlink
  // spinner.text = "Restoring symlink"
  // await fs.remove(contentPath)
  // await fs.symlink(linkTarget, contentPath)

  // Step 10: Assert content is a symlink again
  const finalContentStats = await fs.lstat(contentPath)
  if (!finalContentStats.isSymbolicLink()) {
    throw new Error("content is not a symbolic link after restoration")
  }

  // Step 11: Check if symlink is accessible
  await fs.access(contentPath)
}

function generateContentHash(dir: string): string {
  const files = fs.readdirSync(dir)
  let hash = crypto.createHash("md5")

  for (const file of files) {
    const filePath = path.join(dir, file)
    const stats = fs.statSync(filePath)

    if (stats.isDirectory()) {
      hash.update(generateContentHash(filePath))
    } else {
      hash.update(`${file}:${stats.mtime.getTime()}`)
    }
  }

  return hash.digest("hex")
}

async function checkAndSync() {
  updateSpinner("Checking for changes...")

  const currentHash = generateContentHash(contentPath)
  lastCheckTime = new Date()

  if (currentHash !== previousHash) {
    updateSpinner("Changes detected, syncing content...")
    try {
      await main()
      lastPublishTime = new Date()
      publishStatus = "Success"
      persistMessage("Content synced successfully")
    } catch (error) {
      publishStatus = "Failed"
      persistMessage(`Content sync failed ${error}`, true)
    }
    previousHash = currentHash
  }

  updateDisplay()
}

function updateDisplay() {
  const checkTime = lastCheckTime.toLocaleTimeString()
  const publishTime = lastPublishTime ? lastPublishTime.toLocaleTimeString() : "N/A"

  spinner.text = `
⌜------------------⌝
| Last Check    🕵️ : ${previousHash ? "Changes Detected" : "No Changes"}  ${checkTime}
|------------------:
| Last Publish  🚀 : ${publishStatus}  ${publishTime}
⌞------------------⌟
`
}

function currentBranch() {
  return execCommand("git rev-parse --abbrev-ref HEAD")
}

const checkInterval = setInterval(checkAndSync, checkIntervalMs)

process.on("SIGINT", () => {
  clearInterval(checkInterval)
  spinner.stop()
  console.log("\nScript terminated by user")
  process.exit(0)
})

checkAndSync()

[quakeboy]

Same happens to me too, I assumed git would handle it as a folder and copy the contents. Seems like I am wrong. What could be an easy solution for an ex-developer (but not web)?

I want something where I can update my github repo from either my desktop or macbook. I already put the contents on icloud drive, so I have the obsidian vaults in both computers.

Is there a different way?