Unsupported DB type Tokyo Cabinet, compiled for tokyocabinet

[AGI-chandler]

I'd like to copy the databases generated on our various computers to a webserver, so they could all be viewed with the cgi command, but I'm running into this error even when trying to read the database with simple info command. It doesn't make much sense but I imagine it has something to do with incompatibility of libraries or something.

For example, I downloaded the 1.4.5 release and compiled/installed it on an old computer that has these dependencies:

pango-devel         1.28.1-11.el6
cairo-devel         1.8.8-6.el6_6
tokyocabinet-devel  1.4.33-6.el6

Then, a database was generated with simply duc index /path and transferred to the webserver, which also has duc 1.4.5 compiled from the source, and these dependencies:

libncursesw5-dev     6.2+20201114-2
libpango1.0-dev      1.46.2-3
libcairo2-dev        1.16.0-5
libtokyocabinet-dev  1.4.48-13

When I tried a simple info command, I got the error:

# duc info -d remote-path.db 
Error opening: remote-path.db - unsupported DB type Tokyo Cabinet, compiled for tokyocabinet
Unknown error, contact the author
# 

I figured since the dependencies on the destination webserver are newer, they would be compatible with the older. That doesn't seem to be the case, so I even tried to compile latest Tokyo Cabinet 1.4.48 on the source machine and then rebuild duc with that, but I'm still getting the same error when trying to open the database on the webserver!

[zevv]

Short version: tokyo cabinet is not nice, if you need cross platform compat, you are probably better off with one of the other back ends.

Long version: Yeah, Tokyo Cabinet has been giving us a lot of headaches; it's not stable, it's unmaintained, not compatible over platforms and generally just kind of crappy. What it's great at though, is creating small database files, which is why it is still the default for Duc. Given the amount of problems we have seen in the past - and still have - maybe it is time to consider moving to another default back end somehow; I have no clue yet how to properly support backwards compatibility for existing users though....

[AGI-chandler]

hmm... so that mean I need to compile duc manually on all machines, and configure --with-db-backend? Which would you recommend? leveldb, sqlite3, lmdb, kyotocabinet

[l8gravely]

"Chandler" == Chandler @.***> writes:

hmm... so that mean I need to compile duc manually on all machines, and configure --with-db-backend? Which would you recommend? leveldb, sqlite3, lmdb, kyotocabinet

I would NOT recommend sqlite3, it's not really the best. If I look in the docs, specifically the INSTALL file, it gives a small table showing relative times and sizes.

I'd probably go with leveldb as my next choice.

John

[AGI-chandler]

Thanks @l8gravely I see what you're talking about. I see also it mentions the issue with interoperability between machines there too, but recommends sqlite! However I would also agree with you to not make that my next choice. I'll see how it goes with leveldb.

[l8gravely]

"Chandler" == Chandler @.***> writes:

Thanks @l8gravely I see what you're talking about. I see also it mentions the issue with interoperability between machines there too, but recommends sqlite! However I would also agree with you to not make that my next choice. I'll see how it goes with leveldb.

The other question is exactly how different your two systems are? And would it be possible to maybe just run the web server on the same host you do the scanning from? That's what I do, since it's really just me and other IT people looking at the web pages, though I can and have sent others the link to look at stuff.

Heh, my main duc host is still RHEL 6.6 with:

duc --version duc version: 1.4.3 options: cairo x11 ui tokyocabinet

[AGI-chandler]

They aren't that different, binaries from one would probably run on the other.  They're both Intel-based 64-bit.  The webserver is Debian 11 though and other system is Centos 6.10 I think.  Glad to know I'm not the only one still running this stanky old OS.  It's not like I want to, but I inherited some of these systems and Red Hat doesn't allow their OS to be upgraded.  If I ever can find the time, since I have to reinstall the OS anyway, I'll be putting Debian on there too.

This is only one system though of about 10 that I want to have stat's for, installing a webserver on all of them just isn't practical, plus I'd have to get NetOps involved to get firewalled ports open.  Transferring all the databases to one central webserver would be better, hopefully I find a database that works between systems.

[l8gravely]

"Chandler" == Chandler @.***> writes:

They aren't that different, binaries from one would probably run on the other.  They're both Intel-based 64-bit.  The webserver is Debian 11 though and other system is Centos 6.10 I think. 

That's the problem. Each of those systems is pretty damn close, but if you look at the README for duc you'll see how different they are in packages and names and what needs to be installed just to build duc on them.

Glad to know I'm not the only one still running this stanky old OS.  It's not like I want to, but I inherited some of these systems and Red Hat doesn't allow their OS to be upgraded.  If I ever can find the time, since I have to reinstall the OS anyway, I'll be putting Debian on there too.

I've been moving to Ubuntu Server for alot of stuff. I do like Debian Buster and such, but the pace of change (especially for desktop stuff, not as big a concern for server stuff) was slow enough that it got painful.

This is only one system though of about 10 that I want to have stat's for, installing a webserver on all of them just isn't practical, plus I'd have to get NetOps involved to get firewalled ports open.  Transferring all the databases to one central webserver would be better, hopefully I find a database that works between systems.

Yeah, I used to run my master data collection job on one host and depending on the DB I wanted, I had to go across country to a remote site to scan and then copy the DB(s) back. But I also had the same base OS everywhere. If anything, I'd spend my time getting that part fixed.

Maybe you can do scanning and dumping to a text file 'duc info ...' to get back info. Unless these are huge systems, maybe getting a decent upper level look that goes down four levels is a good compromise?

But if they're not that big, maybe lmdb or leveldb would work. Shouldn't be hard to test. I'd probably try to compile it on your CentOS 6.x box and then try to move the DB to Debian, but I'd look carefully at the package versions of the libs and go with the oldest one, since I'd hope that older versions were forwards compatible.

Good luck! John

[AGI-chandler]

Thanks!

hmm LevelDB isn't working quite right either, but it definitely works better than Tokyo Cabinet in that I can get the CGI script executed and graph displayed, at least.  I turned on tooltips and in the tips I am seeing a lot of "Requested path not found" and in the Apache error.log entries such as:

IO error: lock /var/db/duc/srv1-data.ldb/LOCK: Resource temporarily unavailable
Error opening: /var/db/duc/srv1-data.ldb - unsupported DB type leveldb, compiled for leveldb

Same thing is happening with LMDB, and this even happens with the local webserver's database, so it's not an issue of transferring databases.... so will have to try Sqlite unfortunately.

Also when I click on the graph, it's correctly finding the right path to change to, but instead of updating the webpage, it just prints the headers, like:

 Requested path not found /boot

Status: 302 Found Location: ?path=/boot/grub URI: ?path=/boot/grub Connection: close Content-type: text/html 

Further, I found that cmd-cgi.c seems to be printing extraneous "Content-Type: text/html" that doesn't really need to be there.  I thought it was coming from my cgi script for the longest time!!  Then I added additional duc cgi lines and noticed the "Content-Type" was being printed multiple times, and it's in the tooltips too... I commented out lines 273-4, 324, 446-7, 500, and 510 to get rid of that being printed and seems to be functioning the same.

[l8gravely]

"Chandler" == Chandler @.***> writes:

hmm LevelDB isn't working quite right either, but it definitely works better than Tokyo Cabinet in that I can get the CGI script executed and graph displayed, at least.  I turned on tooltips and in the tips I am seeing a lot of "Requested path not found" and in the Apache error.log entries such as:

IO error: lock /var/db/duc/srv1-data.ldb/LOCK: Resource temporarily unavailable Error opening: /var/db/duc/srv1-data.ldb - unsupported DB type leveldb, compiled for leveldb

Interesting, which version of duc are you running? Can you show me the output of 'duc --version' please? This smells like a bug, so I'd like to try and track it down if we can.

Same thing is happening with LMDB, and this even happens with the local webserver's database, so it's not an issue of transferring databases.... so will have to try Sqlite unfortunately.

Hmm... more details on your OS as well that you're running this on. Have you tried running with '--debug' as well on both the index and cgi parts?

Also when I click on the graph, it's correctly finding the right path to change to, but instead of updating the webpage, it just prints the headers, like:

Requested path not found /boot

Status: 302 Found Location: ?path=/boot/grub URI: ?path=/boot/grub Connection: close Content-type: text/html

Funky.

Further, I found that cmd-cgi.c seems to be printing extraneous "Content-Type: text/html" that doesn't really need to be there.  I thought it was coming from my cgi script for the longest time!!  Then I added additional duc cgi lines and noticed the "Content-Type" was being printed multiple times, and it's in the tooltips too... I commented out lines 273-4, 324, 446-7, 500, and 510 to get rid of that being printed and seems to be functioning the same.

Can you open a new issue for this and send your patch so we can take a look at it and figure out what's needed.

But please make sure you're using version 1.4.5 (or newer) so we can make sure this bug is really a bug.

The more details, the better!

Thanks, John

[AGI-chandler]

Interesting, which version of duc are you running?  Can you show me the output of 'duc --version' please?  This smells like a bug, so I'd like to try and track it down if we can.

I've been using 1.4.5 in all my testing:

$ duc --version
duc version: 1.4.5
options: cairo ui sqlite3

I already recompiled it to try with sqlite but I imagine before it said leveldb instead...

Hmm... more details on your OS as well that you're running this on. Have you tried running with '--debug' as well on both the index and cgi parts?

Debian 11 with Linux 5.10.0-20-amd64 and all upgrade installed.  No I will try using --debug and let you know.

Funky.

Huh you don't see that anywhere?

Can you open a new issue for this and send your patch so we can take a look at it and figure out what's needed. But please make sure you're using version 1.4.5 (or newer) so we can make sure this bug is really a bug. The more details, the better! Thanks, John

Okie I'll try to, time is limited.  It's strange no one else has reported it yet.  I've tried different browsers and different webservers.  Even does it on my phone

Here's v1.4.5 compiled from the source release running on old CentOS 6 webserver (it's the only public facing system we have), with sqlite-devel v3.7.17-8.el6 from okay repo.

$ duc --version
duc version: 1.4.5
options: cairo ui sqlite3

Click to expand

# duc index --debug -bx -d /var/db/duc/Ava-boot.sql /boot

``` Writing to database "/var/db/duc/Ava-boot.sql" >> /boot >> grub 126116 128000 stage2 12636 14336 fat_stage1_5 13980 15360 xfs_stage1_5 11972 13312 minix_stage1_5 2180 3072 grub.conf 11780 13312 ffs_stage1_5 > 11 1024 menu.lst 1341 3072 splash.xpm.gz 13396 15360 e2fs_stage1_5 63 2048 device.map 11772 13312 iso9660_stage1_5 12040 13312 ufs2_stage1_5 11380 13312 vstafs_stage1_5 14428 16384 reiserfs_stage1_5 512 2048 stage1 13284 14336 jfs_stage1_5 << grub actual:256891 apparent:283648 216357 217088 symvers-2.6.32-754.35.1.el6.x86_64.gz >> lost+found << lost+found actual:0 apparent:13312 5852456 5853184 initrd-2.6.32-642.11.1.el6.x86_64kdump.img 2658467 2659328 System.map-2.6.32-754.17.1.el6.x86_64 216292 217088 symvers-2.6.32-754.14.2.el6.x86_64.gz 216276 217088 symvers-2.6.32-754.12.1.el6.x86_64.gz 25167890 25168896 initramfs-2.6.32-754.23.1.el6.x86_64.img 4336848 4337664 vmlinuz-2.6.32-754.35.1.el6.x86_64 25127503 25127936 initramfs-2.6.32-754.12.1.el6.x86_64.img 108308 108544 config-2.6.32-754.14.2.el6.x86_64 171 1024 .vmlinuz-2.6.32-754.17.1.el6.x86_64.hmac 4335472 4335616 vmlinuz-2.6.32-754.14.2.el6.x86_64 4331824 4332544 vmlinuz-2.6.32-754.12.1.el6.x86_64 108308 108544 config-2.6.32-754.17.1.el6.x86_64 25129519 25129984 initramfs-2.6.32-754.14.2.el6.x86_64.img 171 1024 .vmlinuz-2.6.32-754.12.1.el6.x86_64.hmac 171 1024 .vmlinuz-2.6.32-754.23.1.el6.x86_64.hmac 4335696 4336640 vmlinuz-2.6.32-754.17.1.el6.x86_64 >> efi >> EFI >> redhat 254317 254976 grub.efi << redhat actual:254317 apparent:257024 << EFI actual:254317 apparent:259072 << efi actual:254317 apparent:261120 216292 217088 symvers-2.6.32-754.17.1.el6.x86_64.gz 25129468 25129984 initramfs-2.6.32-754.17.1.el6.x86_64.img 2658242 2658304 System.map-2.6.32-754.14.2.el6.x86_64 4939930 4940800 initrd-2.6.32-573.el6.x86_64kdump.img 108308 108544 config-2.6.32-754.12.1.el6.x86_64 2657436 2658304 System.map-2.6.32-754.12.1.el6.x86_64 2658899 2659328 System.map-2.6.32-754.23.1.el6.x86_64 216292 217088 symvers-2.6.32-754.23.1.el6.x86_64.gz 108308 108544 config-2.6.32-754.35.1.el6.x86_64 171 1024 .vmlinuz-2.6.32-754.35.1.el6.x86_64.hmac 25168802 25169920 initramfs-2.6.32-754.35.1.el6.x86_64.img 108308 108544 config-2.6.32-754.23.1.el6.x86_64 171 1024 .vmlinuz-2.6.32-754.14.2.el6.x86_64.hmac 5849455 5850112 initrd-2.6.32-642.6.1.el6.x86_64kdump.img 2660415 2661376 System.map-2.6.32-754.35.1.el6.x86_64 4335888 4336640 vmlinuz-2.6.32-754.23.1.el6.x86_64 << /boot actual:179469322 apparent:179543040 Indexed 50 files and 6 directories, (179469322B apparent, 179543040B actual) in 0.00 secs. # ```

Don't see any issues there.

Then, in my ~/public_html/cgi-bin/ dir I put ava.sh: (click to expand)

``` #!/bin/bash DUC="/usr/local/bin/duc" CAT="/bin/cat" echo "Content-type: text/html" echo "" ${CAT} << § § ${DUC} cgi --debug --list --tooltip -l 4 -d /var/db/duc/Ava-boot.sql /boot ${CAT} << §

§ ```

You can view it here: https://ava.genome.arizona.edu/~scar/cgi-bin/ava.sh I'm not seeing any "Requested path not found" errors or database access errors either, so that's nice... the only thing in httpd's error.log is Reading from database "/var/db/duc/Ava-boot.sql" repeated over and over. However, the problem of "Content-Type: text/html" showing up everywhere is evident.  It's in every tooltip, for example. Further, clicking on the graph doesn't do anything. New problem with this instance is that the graph is only one level even though it's supposed to be 4 levels by default but I explicitly added it as an option too.

[l8gravely]

"Chandler" == Chandler @.***> writes:

Interesting, which version of duc are you running?  Can you show me the output of 'duc
--version' please?  This smells like a bug, so I'd like to try and track it down if we can.

I've been using 1.4.5 in all my testing:

$ duc --version duc version: 1.4.5 options: cairo ui sqlite3

Good.

I already recompiled it to try with sqlite but I imagine before it said leveldb instead...

Can you confirm that you're using the same version of leveldb I've got here? My main home machine is Debian 11 as well, so it should be trivial to try and do testing here. I admit I'm also running kernel 6.1.x that I self compile...

 $ dpkg-query -l |grep leveldb
 ii  leveldb-doc                                              1.22-3
 all          LevelDB documentation
 ii  libleveldb-dev:amd64                                     1.22-3
 amd64        fast key-value storage library (development files)
 ii  libleveldb1d:amd64                                       1.22-3
 amd64        fast key-value storage library
 rc  libleveldb1v5:amd64                                      1.18-5
 amd64        fast key-value storage library

Hmm... more details on your OS as well that you're running this on. Have you tried running
with '--debug' as well on both the index and cgi parts?

Debian 11 with Linux 5.10.0-20-amd64 and all upgrade installed.  No I will try using --debug and let you know.

Funky.

Huh you don't see that anywhere?

Can you open a new issue for this and send your patch so we can take a look at it and figure
out what's needed. But please make sure you're using version 1.4.5 (or newer) so we can make
sure this bug is really a bug. The more details, the better! Thanks, John

Okie I'll try to, time is limited.  It's strange no one else has reported it yet.  I've tried different browsers and different webservers.  Even does it on my phone

Here's v1.4.5 compiled from the source release running on old CentOS 6 webserver (it's the only public facing system we have), with sqlite-devel v3.7.17-8.el6 from okay repo.

$ duc --version duc version: 1.4.5 options: cairo ui sqlite3

Click to expand

duc index --debug -bx -d /var/db/duc/Ava-boot.sql /boot

Writing to database "/var/db/duc/Ava-boot.sql"

/boot grub 126116 128000 stage2 12636 14336 fat_stage1_5 13980 15360 xfs_stage1_5 11972 13312 minix_stage1_5 2180 3072 grub.conf 11780 13312 ffs_stage1_5 11 1024 menu.lst 1341 3072 splash.xpm.gz 13396 15360 e2fs_stage1_5 63 2048 device.map 11772 13312 iso9660_stage1_5 12040 13312 ufs2_stage1_5 11380 13312 vstafs_stage1_5 14428 16384 reiserfs_stage1_5 512 2048 stage1 13284 14336 jfs_stage1_5 << grub actual:256891 apparent:283648 216357 217088 symvers-2.6.32-754.35.1.el6.x86_64.gz lost+found << lost+found actual:0 apparent:13312 5852456 5853184 initrd-2.6.32-642.11.1.el6.x86_64kdump.img 2658467 2659328 System.map-2.6.32-754.17.1.el6.x86_64 216292 217088 symvers-2.6.32-754.14.2.el6.x86_64.gz 216276 217088 symvers-2.6.32-754.12.1.el6.x86_64.gz 25167890 25168896 initramfs-2.6.32-754.23.1.el6.x86_64.img 4336848 4337664 vmlinuz-2.6.32-754.35.1.el6.x86_64 25127503 25127936 initramfs-2.6.32-754.12.1.el6.x86_64.img 108308 108544 config-2.6.32-754.14.2.el6.x86_64 171 1024 .vmlinuz-2.6.32-754.17.1.el6.x86_64.hmac 4335472 4335616 vmlinuz-2.6.32-754.14.2.el6.x86_64 4331824 4332544 vmlinuz-2.6.32-754.12.1.el6.x86_64 108308 108544 config-2.6.32-754.17.1.el6.x86_64 25129519 25129984 initramfs-2.6.32-754.14.2.el6.x86_64.img 171 1024 .vmlinuz-2.6.32-754.12.1.el6.x86_64.hmac 171 1024 .vmlinuz-2.6.32-754.23.1.el6.x86_64.hmac 4335696 4336640 vmlinuz-2.6.32-754.17.1.el6.x86_64 efi EFI redhat 254317 254976 grub.efi << redhat actual:254317 apparent:257024 << EFI actual:254317 apparent:259072 << efi actual:254317 apparent:261120 216292 217088 symvers-2.6.32-754.17.1.el6.x86_64.gz 25129468 25129984 initramfs-2.6.32-754.17.1.el6.x86_64.img 2658242 2658304 System.map-2.6.32-754.14.2.el6.x86_64 4939930 4940800 initrd-2.6.32-573.el6.x86_64kdump.img 108308 108544 config-2.6.32-754.12.1.el6.x86_64 2657436 2658304 System.map-2.6.32-754.12.1.el6.x86_64 2658899 2659328 System.map-2.6.32-754.23.1.el6.x86_64 216292 217088 symvers-2.6.32-754.23.1.el6.x86_64.gz 108308 108544 config-2.6.32-754.35.1.el6.x86_64 171 1024 .vmlinuz-2.6.32-754.35.1.el6.x86_64.hmac 25168802 25169920 initramfs-2.6.32-754.35.1.el6.x86_64.img 108308 108544 config-2.6.32-754.23.1.el6.x86_64 171 1024 .vmlinuz-2.6.32-754.14.2.el6.x86_64.hmac 5849455 5850112 initrd-2.6.32-642.6.1.el6.x86_64kdump.img 2660415 2661376 System.map-2.6.32-754.35.1.el6.x86_64 4335888 4336640 vmlinuz-2.6.32-754.23.1.el6.x86_64 << /boot actual:179469322 apparent:179543040 Indexed 50 files and 6 directories, (179469322B apparent, 179543040B actual) in 0.00 secs.

Don't see any issues there.

Then, in my ~/public_html/cgi-bin/ dir I put ava.sh: (click to expand)

!/bin/bash

DUC="/usr/local/bin/duc" CAT="/bin/cat"

echo "Content-type: text/html" echo ""

${CAT} << § <!DOCTYPE html>

§ ${DUC} cgi --debug --list --tooltip -l 4 -d /var/db/duc/Ava-boot.sql /boot ${CAT} << §

§

You can view it here: https://ava.genome.arizona.edu/~scar/cgi-bin/ava.sh I'm not seeing any "Requested path not found" errors or database access errors either, so that's nice... the only thing in httpd's error.log is Reading from database "/var/db/duc/Ava-boot.sql" repeated over and over. However, the problem of "Content-Type: text/html" showing up everywhere is evident.  It's in every tooltip, for example. Further, clicking on the graph doesn't do anything. New problem with this instance is that the graph is only one level even though it's supposed to be 4 levels by default but I explicitly added it as an option too.

Ok, so we've got two issues here. First is the leveldb possible problem. The second is the "Content-Type: text/html" issue, which should be quicker and easier to fix.

I'll try to find some time to poke at this and see what I can do.

If you can come up with a small test case for leveldb, that would be awesome. I'll try some stuff too.

John

[l8gravely]

"Chandler" == Chandler @.***> writes:

Then, in my ~/public_html/cgi-bin/ dir I put ava.sh: (click to expand)

!/bin/bash

DUC="/usr/local/bin/duc" CAT="/bin/cat"

echo "Content-type: text/html" echo ""

${CAT} << § <!DOCTYPE html>

§ ${DUC} cgi --debug --list --tooltip -l 4 -d /var/db/duc/Ava-boot.sql /boot ${CAT} << §

§

You can view it here: https://ava.genome.arizona.edu/~scar/cgi-bin/ava.sh

Ok, so I've looked at this and I think the problem is that you are having is that your ava.sh script is doing the Content-type: header, when it shouldn't be at all. All I have in my user.cgi (like your ava.sh) script is the following:

!/bin/sh

args="--tooltip -d /data//duc/user.db --palette=rainbow --list --header=header.htm --footer=footer.htm"

/usr/local/bin/duc cgi $args

And all the HTML headers are generated by 'duc cgi ...' so you don't need to do them at all. So remove all your ${CAT} stuff completely.

John

[AGI-chandler]

all the HTML headers are generated by 'duc cgi ...' so you don't need to do them at all.

Okay, I thought I tried that when I first set it up but got some sort of Internal Server error from httpd.  Since I've never written or run CGI scripts before, then I found some examples online and that's what ava.sh was based on.  Also, according to help cgi there should be a [PATH] at the end, but I guess that's optional since you didn't include one.  Maybe that's where all the "Requested path not found" errors were coming from.  So ava.sh now just has:

$ cat ava.sh
#!/bin/sh
DUC="/usr/local/bin/duc"
ARGS1="cgi --css-url=/~scar/ava.css -d /var/db/duc/Ava-boot.sql --list --tooltip"

${DUC} ${ARGS1}

So that got rid of all the "Content-type" being printed but there are still these issues:

1. clicking on the graph doesn't do anything.
2. graph only shows one level, no directories.  if I open "efi" from the list then can see that it shows dirs and multiple levels.

[AGI-chandler]

Also I noticed the following:

1. with --css-url=/~scar/ava.css added to the args (it just sets the font style to mono for now), then the tooltips stop working right and instead are shown at the bottom of the file list (notice the scrollbar change length, or try zooming out so everything fits on the screen). Maybe ava.css is missing something, I don't know much about web code...
2. When viewing the "efi" folder for example, then clicking on that graph works as expected...

[l8gravely]

"Chandler" == Chandler @.***> writes:

all the HTML headers are generated by 'duc cgi ...' so you don't need to do them at all.

Okay, I thought I tried that when I first set it up but got some sort of Internal Server error from httpd.  Since I've never written or run CGI scripts before, then I found some examples online and that's what ava.sh was based on.  Also, according to help cgi there should be a [PATH] at the end, but I guess that's optional since you didn't include one.  Maybe that's where all the "Requested path not found" errors were coming from.  So ava.sh now just has:

$ cat ava.sh

!/bin/sh

DUC="/usr/local/bin/duc" ARGS1="cgi --css-url=/~scar/ava.css -d /var/db/duc/Ava-boot.sql --list --tooltip"

${DUC} ${ARGS1}

So that got rid of all the "Content-type" being printed but there are still these issues:

Good.

1. clicking on the graph doesn't do anything.

Hmm... try running the CGI script with the --debug option and send us the output to look over.

Maybe build a DB of your home directory as a test (don't need to put it on the internet) or maybe download the linux kernel source and extra that and run duc over that as a test.

2. graph only shows one level, no directories.  if I open "efi" from the list then can see that it shows dirs and multiple levels.

What do you mean "open efi from the list" and then it works? It almost sounds like you have something messed up in your DB and how you indexed things.

But I went to the URL you posted and I see what you're talking about. But bizarelly, the directories are being listed below, not on the side of the image. I'd have to login to $WORK to see how I setup things to make them work. Ah wait, I see you included your own .css file, so ignore that part of my question.

So from looking at your web page, I think you indexed /boot/efi seperately from /boot. And honestly I don't like combining seperate fileststems into one DB because of problems like this.

I bet /boot, /boot/efi and /boot/grub and /boot/lost+found are all

Can you blow away the entire index file Ava-boot.sql and re-index with:

duc index -d /var/db/duc/Ava-boot.sql /boot

and then do:

duc info -d /var/db/duc/Ava-boot.sql /boot

and then do:

duc ls -R -d /var/db/duc/Ava-boot.sql /boot

and send the output.

John

[AGI-chandler]

So from looking at your web page, I think you indexed /boot/efi seperately from /boot.

No I just ran the one command.

I think the problem though was due to the CSS I was using, it was too basic.  I copied the CSS from your code and then changed the font-family to monospace, now all the problems went away, amazing!  Told you I didn't know much about web code, ha!  Can't believe styles can cause so many problems.

I re-indexed /boot as you requested:

``` # rm /var/db/duc/Ava-boot.sql rm: remove regular file `/var/db/duc/Ava-boot.sql'? y # duc index -d /var/db/duc/Ava-boot.sql /boot # duc info -d /var/db/duc/Ava-boot.sql /boot Date Time Files Dirs Size Path 2023-02-17 00:58:04 50 6 171.2M /boot # duc ls -R -d /var/db/duc/Ava-boot.sql /boot 24.0M ╰┬─ initramfs-2.6.32-754.35.1.el6.x86_64.img 24.0M ├─ initramfs-2.6.32-754.23.1.el6.x86_64.img 24.0M ├─ initramfs-2.6.32-754.14.2.el6.x86_64.img 24.0M ├─ initramfs-2.6.32-754.17.1.el6.x86_64.img 24.0M ├─ initramfs-2.6.32-754.12.1.el6.x86_64.img 5.6M ├─ initrd-2.6.32-642.11.1.el6.x86_64kdump.img 5.6M ├─ initrd-2.6.32-642.6.1.el6.x86_64kdump.img 4.7M ├─ initrd-2.6.32-573.el6.x86_64kdump.img 4.1M ├─ vmlinuz-2.6.32-754.35.1.el6.x86_64 4.1M ├─ vmlinuz-2.6.32-754.23.1.el6.x86_64 4.1M ├─ vmlinuz-2.6.32-754.17.1.el6.x86_64 4.1M ├─ vmlinuz-2.6.32-754.14.2.el6.x86_64 4.1M ├─ vmlinuz-2.6.32-754.12.1.el6.x86_64 2.5M ├─ System.map-2.6.32-754.35.1.el6.x86_64 2.5M ├─ System.map-2.6.32-754.23.1.el6.x86_64 2.5M ├─ System.map-2.6.32-754.17.1.el6.x86_64 2.5M ├─ System.map-2.6.32-754.14.2.el6.x86_64 2.5M ├─ System.map-2.6.32-754.12.1.el6.x86_64 277.0K ├─ grub 125.0K │ ╰┬─ stage2 16.0K │ ├─ reiserfs_stage1_5 15.0K │ ├─ xfs_stage1_5 15.0K │ ├─ e2fs_stage1_5 14.0K │ ├─ jfs_stage1_5 14.0K │ ├─ fat_stage1_5 13.0K │ ├─ ufs2_stage1_5 13.0K │ ├─ minix_stage1_5 13.0K │ ├─ ffs_stage1_5 13.0K │ ├─ iso9660_stage1_5 13.0K │ ├─ vstafs_stage1_5 3.0K │ ├─ grub.conf 3.0K │ ├─ splash.xpm.gz 2.0K │ ├─ stage1 2.0K │ ├─ device.map 1.0K │ ╰─ menu.lst 255.0K ├─ efi 253.0K │ ╰─ EFI 251.0K │ ╰─ redhat 249.0K │ ╰─ grub.efi 212.0K ├─ symvers-2.6.32-754.35.1.el6.x86_64.gz 212.0K ├─ symvers-2.6.32-754.14.2.el6.x86_64.gz 212.0K ├─ symvers-2.6.32-754.17.1.el6.x86_64.gz 212.0K ├─ symvers-2.6.32-754.23.1.el6.x86_64.gz 212.0K ├─ symvers-2.6.32-754.12.1.el6.x86_64.gz 106.0K ├─ config-2.6.32-754.12.1.el6.x86_64 106.0K ├─ config-2.6.32-754.14.2.el6.x86_64 106.0K ├─ config-2.6.32-754.17.1.el6.x86_64 106.0K ├─ config-2.6.32-754.23.1.el6.x86_64 106.0K ├─ config-2.6.32-754.35.1.el6.x86_64 13.0K ├─ lost+found 1.0K ├─ .vmlinuz-2.6.32-754.12.1.el6.x86_64.hmac 1.0K ├─ .vmlinuz-2.6.32-754.14.2.el6.x86_64.hmac 1.0K ├─ .vmlinuz-2.6.32-754.17.1.el6.x86_64.hmac 1.0K ├─ .vmlinuz-2.6.32-754.23.1.el6.x86_64.hmac 1.0K ╰─ .vmlinuz-2.6.32-754.35.1.el6.x86_64.hmac # ```

I also tested with linux source code.  Is it possible to have multiple duc cgi commands in one page/script?  I thought that would be nice because it starts with the info printout.  But, when I tried that it doesn't work, e.g.:

script (no custom CSS):

``` #!/bin/sh DUC="/usr/local/bin/duc" CGI_BOOT="cgi -d /var/db/duc/Ava-boot.sql --list --tooltip" CGI_LINUX="cgi -d /var/db/duc/linux.sql --list --tooltip" ${DUC} ${CGI_BOOT} ${DUC} ${CGI_LINUX} ```

output:

 Notice the extra `Content-Type: text/html` is showing up again, and `/tmp/linux` doesn't render at all (but /boot still does):  Now if I take out the `${DUC} ${CGI_BOOT}` line everything works as expected, see for yourself: https://ava.genome.arizona.edu/~scar/cgi-bin/ava.sh

Also I've copied some sqlite3 databases from other systems compiled with duc 1.4.5 and those are displaying fine as well.  Now that I've figured out the quirks, I think this will be very helpful in organizing all our data, over 100 million files and a petabyte!  Thanks so much!!

[AGI-chandler]

DUC has been working well so far, I have about half our data indexed.  I've transferred the databases to our internal webserver and the graphs and everything are displaying and working correctly!  Very happy with it so I'll go ahead and close this.  Thanks everyone!