Remarks config

[anbzerc]

Hi, sorry to bother you again. I don't understand how to install and config remarks to use it with scrybble :/. Would you have some precisions on the docker command to run ? Thank you

[Azeirah]

Hey, as is shown in the Readme I'm not sure yet. The deployed version of scrybble runs natively on the OS, so it can call docker containers.

The dockerized version cannot do this, so I need to think of a solution.

During development, I run remarks manually from the host OS, that is definitely possible. You can run it by installing the project and its dependencies using python, or you can use the docker container.

There's an example command for running the container in the Readme. Your notebooks can be found in storage/efs/user-{user-id}.

[anbzerc]

Hi, I have a question, if I have highlights on a book, there should be a xochilt folder uuid.highlights ?

[Azeirah]

Sometimes. I'm not sure from the top of my head what highlights type that file is for, smart highlights are specified in the .rm file iirc.

Remarks supports both.

[Azeirah]

I think this approach could work:

https://www.reddit.com/r/docker/comments/11qejqg/is_there_any_way_to_transparently_use_a/

• add container with remarks, run netcat listening for commands on a port, forward the commands to remarks
• make a volume for storage/efs, share this volume between the app and the remarks container
• send commands to remarks nectat port from laravel app instead of running the docker command if env is local

I think that's pretty easy to implement.

[Azeirah]

I'm making progress, I think this is in the right direction

Write a dockerfile for remarks as a server

FROM laauurraaa/remarks-bin:0.3.16

RUN apt-get update && apt-get install netcat -yqq

COPY listen.sh /listen.sh

RUN ["chmod", "+x", "/listen.sh"]
RUN ["mknod", "-m", "777", "/fifo", "p"]

ENTRYPOINT ["bash", "/listen.sh"]

Write a server script for remarks

This is listen.sh

while true; do
    cat /fifo | nc -lkp 1234 | while read line; do
        IFS=' ' read -ra args <<< "$line"
        bash -c "/root/.local/bin/poetry run python -m remarks ${line}" &> /fifo
    done
done

^ This script is a bit wonky. As long as the container is not exposed to the internet it shouldn't be too much of a problem, but given that the filenames come from the remarkable tablet folder structure it is vulnerable to injections as it's written right now. Need to sanitize the input somehow, the remarks container has access to the entire docker network as it is right now soooo.. rather not

Add to docker-compose

...
    remarks:
        build:
            context: ./docker/remarks-server
            dockerfile: Dockerfile
        networks:
            - sail

Call remarks from the app container

It's now possible to connect to the remarks server from the app container

echo YOUR REMARKS PARAMS HERE | nc remarks 1234

All that's left is to create a shared volume between the app and the remarks containers giving the remarks container access to downloaded notebooks

And write a "RemarksServerService" php implemenation of the RemarksService class that already exists. All this needs to do is send the right params to the remarks server container on port 1234 with curl or whatever TCP client PHP has and wait for remarks to terminate.

(extract interface, depend on interface instead of RemarksService, implement interface in RemarksServerService, configure which service to put into the DI container in kernel.php depending on app env)

[Azeirah]

It's probably best not to use netcat tbh, it does "work" but I'll just write a simple flask server instead. It's a lot easier to work with than bash when it comes to processing the request, making sure the params are sanitized, handling multiple connections, clearer error logging etc :p

[Azeirah]

@anbzerc I wrote the flask server and added a simple local remarks docker container.

I thought that'd be it but it still won't work locally 100% since it now tries to create a temporary link to make the processed file available to the Obsidian plugin by uploading it to Amazon's S3 file storage.

Needs a similar strategy to remarks, if prod use s3, if local then just serve the file from the filesystem.

The ProcessDownloadedZipController's 6th step does this right now.

        // 6. Upload zip to S3
        $sync_context->logStep("Uploading zip to storage");
        $s3_download_path = "userZips/{$sync_context->sync_id}.zip";
        if (!Storage::disk('s3')->put($s3_download_path, $user_storage->get($to1))) {
            $sync_context->logStep("Failed to upload zip to storage");
            throw new RuntimeException('Unable to upload zip to s3');
        }
        $sync_context->logStep("Uploaded zip to storage");

All that needs to be done is abstract this logic into an interface and implement two strategies, one for production and one for local, just like I did for the remarks runner.

As for the local implementation, I suppose all it needs to do is move the generated zip to the public folder.

Then there's only one last step remaining, and that's the controller that handles generating a link for the zip file when the Obsidian plugin requests it for download.

This is the SyncController.php.

        $user = Auth::user();

        if ($user === null) {
            throw new UnauthorizedException('You need to be logged in in order to interact with the API');
        }
        $disk = Storage::disk('s3');
        $results =
            Sync::forUser($user)
                ->whereIsCompleted()
                ->get(['filename', 'S3_download_path', 'id'])
                ->map(fn(Sync $sync) => [
                    'download_url' => $disk->temporaryUrl(
                        $sync->S3_download_path,
                        now()->addMinutes(5)),
                    'filename' => $sync->filename,
                    'id' => $sync->id
                ]);

        return response()->json($results);

^ Needs to be put in its own implementation of ObsidianSyncRequestServiceInterface or whatever whacky name.

A version doing approx the same thing but with the url coming from public instead of from S3 needs to be made for local.

Again, abstract logic into interface for generating the URL, with implementations for production and local. Prod is the code currently in the SyncController, local just kinda returns the public path for the correct file based on the S3_download_path property of the Sync model db row...

All sounds complex but it's not very difficult, but I'm just kinda busy with other stuff atm and was hoping it'd be finished after implementing the remarks docker service 😅