Enhancement Idea: Export Log

[STaRDoGG]

While trying to diagnose some issues with a container that spits out a ton of stuff to the log and noticing that once the log scrolls down past a certain point, even going all the way back up wasn't showing all the previous data I thought it might be handy to have a small button, maybe next to the arrow at the bottom right, that lets you export the entire log up until that point, to a text file, to make it easier to review the entire log, and even do things such as upload the log to sites, and so on.

I know that of you have many containers it could potentially be a lot of space used on the device, so maybe include the option to only save the log(s) of the desired container(s) that can easily be toggled on and off. That way if you want to only save the log of say, 1 container, you just toggle it on when you need to do your thing, and then toggle it back off when done doing your thing.

[amir20]

Hey @STaRDoGG. Happy new year!

Can you help me understand your use case a little more?

I think I get what you are saying, but your second paragraph confuses me.

Did you mean...

a. Create an export button which would download the whole log for that container in to something like container_id.log.gz

or

b. just download what is in the browser?

For a, I don't think it would use any space at all in the browser. It would be done all in the API. And for b, I don't know what "That way if you want to only save the log of say, 1 container, you just toggle it on when you need to do your thing, and then toggle it back off when done doing your thing." means. Can you clarify? Why would there be a toggle? Isn't just a download button?

I think it would make most sense to just have a download button which would export the whole log like option a. But I wonder why doing docker logs <name> > file.log isn't a better option? Do you have access to dozzle but not the host?

Dozzle's focus has been mostly for real-time debugging. Going back to logs from a while back might not make sense. But I am not against it. I just want to make sure I understand your use case.

[STaRDoGG]

Heyas @amir20 , HNY as well!

Happy to see how far Dozzle has come along, it's been VERY useful. =)

Regarding your questions:

a) Exactly. Sometimes logs can get so long that they scroll off screen, and even trying to scroll back up only lets so much show before you "lose" what was there; I'd like to be able to export the entire log since the start of the container, to a text file (doesn't necessarily need to be compressed afterward, but could also be a nice option as well). I've created a simple mockup but the implementation is completely up to you, this is just to illustrate the idea a bit:

Clicking the "Save" icon will bring up the usual "save to" dialog where we can choose the folder to save the text file toolbar (unless you just want to have a folder hard coded to automatically save it to). Of course, in order to be able to export the entire log since the container started, Dozzle would have to be storing the log to disk, which leads me to ...

b) In the latter case, I'm just trying to think ahead a bit to how much storage could potentially be used (unwantedly) if Dozzle automatically were to start storing the log to the hard drive for every single container by default; so in order to work around that, just give the option to only save the log(s) of the container(s) that you know you'll want to save, and to do that, just add a toggle button, similar to the "pin" toggle you already have. Here's another basic illustration:

When toggling the "save" icon On, dozzle will only save to disk that container's logs (for the ability to save to the text file later. Really it'll already be saved then, but the Save icon with the ability to "save to" a user's folder choice would let you then send that file to wherever the user chose.).

Update: Thinking a bit more through it now, maybe just toggling on whichever containers you want to save to text would just save them to a "logs" folder with their container name as the file name, and you wouldn't need the button in the first screenshot, as all the logs would just go to one folder. I dunno, I'm just thinking out loud to give you some ideas. :p

Btw, if any previous log was saved to storage for any container, it gets automatically deleted/replaced with the new one, so you only have the log from the start of the container's current session.

Using docker logs <name> > file.log is always an option, but at least in my own opinion, I am using Dozzle to have a GUI to abstract me away from the command-line and have everything right there "in my face" with good ol fashioned mouse click ability. ;) (click a button, save a log file).

Saving the entire log file to text would help real-time debugging in the case of logs that spit out a ton of info. For instance, the LSIO LazyLibrarian container, with the FFMPEG environment variable added. In Dozzle I'm unable to scroll up to the start of the container's session after awhile.

[amir20]

Thanks for the follow up! So there is at least three things to unpack here. Let me try and tell me what you think.

1. For case "a" scenario, I agree that this would be valuable. Since you are debugging, you might want to download the whole file and do some other kind of analysis on it. I think this would be easy to implement. There is currently a memory issue which might make this difficult. If the log size is huge, then it could allocate too much memory. I am doing more research. See https://github.com/amir20/dozzle/pull/915 if interested.

2. I'm just trying to think ahead a bit to how much storage could potentially be used (unwantedly) if Dozzle automatically were to start storing the log to the hard drive for every single container by defau implies that Dozzle needs this to happen. I rely on Docker API to fetch and display logs. I don't actually need to store any logs on disk. Which as it turns out, has made Dozzle a lot simpler. There has been a discussions regarding this. Many people suggested for Dozzle to store all logs and do post processing. I have always pushed back on this because Dozzle is meant to be a real-time log viewer. Not something that can view weeks of logs. I haven't been able to find any simple databases that are simple and low memory. More importantly, because Dozzle doesn't store anything, there are no privacy concerns.

3. Since Dozzle doesn't need to store any logs, just stream them, I have relied heavily on the Docker API. When you scroll up, I try to query for more data. When you said In Dozzle I'm unable to scroll up to the start of the container's session after awhile., that might actually be a bug. I haven't been able to reproduce it though.

So to summarize, Dozzle doesn't need to store any files on disk. It just queries the API. To fetch all logs for a specific container, I can just ask Docker to stream to Dozzle. I think gzip makes sense because with my tests it showed to save 90% disk.

I need to do some more research but for now, I think I'll implement a download button similar to what you suggested on the current container only.

[amir20]

Above is merged which should have download button on top right corner and can be used with docker pull amir20/dozzle:latest-dev.

I'll test it out and release an official version soon. A lot has changed so please test it out @STaRDoGG too.

[STaRDoGG]

Heyas,

I agree with all of your points above, and it all sounds good to me. I haven't ever used the docker API myself (yet) so I wasn't already aware of what you mentioned; that's even better that it makes it super easy. I also agree that it shouldn't really be used to view logs months old; at least not for my own use case.

If you want to try a container where scrolling up was an issue (for me, hopefully it's replicable for you as well) you might try out this config:

services:

      lazylibrarian:
        # https://hub.docker.com/r/linuxserver/lazylibrarian
        image: ghcr.io/linuxserver/lazylibrarian:latest
        hostname: lazylibrarian
        environment:
          - TZ=America/Chicago
          - PUID=1000
          - PGID=1000
          - DOCKER_MODS=linuxserver/mods:lazylibrarian-ffmpeg
        volumes:
          - /opt/docker/configs/Lazylibrarian:/config:rw # Configs
          - /mnt/i/Lazylibrarian/books:/books:rw
          - /mnt/i/Lazylibrarian/downloads:/downloads:rw
          - /etc/localtime:/etc/localtime:ro
        ports:
          - 5299:5299
        deploy:
            mode: replicated
            replicas: 1
            placement:
              constraints:
                - node.labels.MainDaemon == true
            resources:
              limits:
                cpus: '0.5'
                memory: 200M
            restart_policy:
              condition: any
        networks:
          - ocm
          - od

networks:
  ocm:
    external: true
    name: oc-misc
  od:
    external: true
    name: oc-download

This container upon startup shoots out a lot of logs, and after awhile when trying to scroll all the way to the start of the container, Dozzle "gets stuck" with it's loading indicator showing that it's loading, but the older stuff never comes up.

Aside from that, it'd still be useful to be able to export the log (via GUI) for quick searching, pasting, etc.

I will give the latest container a try and report back, thanks!

[STaRDoGG]

Just tested out the latest dev version and so far looks perfect; one question, where did it download the log to? I can't find it. :p

Any chance of adding one of these in there when clicked?

Also, it's no big deal but I was thinking if ya wanted to save a bit of screen real estate/"screen noise", the label "download" isn't even needed; just the icon is plenty. Maybe add "download" as a tooltip if really needed. Just an idea, your call. =)

[amir20]

Any chance of adding one of these in there when clicked?

That depends on your browser. I don't have a Windows machine. But on Mac Firefox and Chrome, I get a download dialog and it saves it in my Downloads directory.

Firefox

Chrome

Also, it's no big deal but I was thinking if ya wanted to save a bit of screen real estate/"screen noise", the label "download" isn't even needed; just the icon is plenty. Maybe add "download" as a tooltip if really needed. Just an idea, your call. =)

I plan to release it and then fix some UI things. It also needs to be hidden when the screen is too small.

[amir20]

v3.2.0 is released.

[amir20]

I tried running ghcr.io/linuxserver/lazylibrarian:latest and it seemed to work for me. Maybe you can share a video with console open to see what kind of errors happen. There is a known issue where if logs are sparse, Dozzle doesn't know how far to go back.

[amir20]

I am closing for now. I haven't found any bugs on the download button.

[STaRDoGG]

Very strange, I originally was thnking it might be because I'm using Free Download Manager to catch all downloads, and sometimes it does act up (and still might be the issue in some respect) so I went into it's settings and set it to not catch downloads, and it still doesn't initiate a download when I click the download button.

So I fired up my Ubuntu WSL2 machine via RDP so I have a GUI and tried it there, where there is no other download manager installed, and it still behaves the same.

I can hold the Shift button when clicking and it does download the file though, so at least it's working in one respect, so I can't really complain, it's just a strange behavior that a regular click doesn't seem to do anything.

I did try opening the inspector and clicking the butrton and it shows this, though I'm not sure whether it really means anything significant:

[amir20]

That's weird. Can you give me specific OS, browser versions you are using? Hopefully something I can spin up on Mac. I also get that error, but it still work.

I am downloading ubuntu to try right now.

[amir20]

I wasn't able to get Ubuntu installed. But used a Linux machine at work and it worked. I also tried Firefox, Safari, Edge and Chrome.

I am guessing something else is going on that I don't know about.

[amir20]

This might fix it. https://github.com/amir20/dozzle/commit/4d4d8f549fcc8020ba678ac590361b92e734b848 I am just guessing though. Try latest-dev again.

[amir20]

@STaRDoGG I brought up Windows and Ubuntu Desktop. Both seemed to work.

[STaRDoGG]

Just tested again and it seems to be working great, I'm not sure if something changed or what, but thanks a million for adding this. =)