Export/import file list

[peps1]

In AirDC++ you can easily export file lists from another user. It would be great to have this feature in the webclient.

This could be implemented in several steps:

1. export plain text information for every file/folder in a single folder within the share
2. export full share information from a user
3. import full share information from another user and browse through web like any normal file list
4. ability to open exported file list that was downloaded from another users share

Personally for me, the first point would be most interesting, as i like to get a list of directories to compare them to my own.

[bingoxo]

any update on this ?

[peps1]

i'm working on adding a command extension to include at least part of this, what exactly of these functions are you looking for?

The plan is to have a command like /list bingoxo /share/folder and it prints a formatted plain text list of that shared folder.. alternatively it would download a text file or write it to a specific log file on disk.

Extension repo: https://github.com/peps1/airdcpp-user-commands

[peps1]

The /list command I mentioned above is now included in the latest version of airdcpp-user-commands extension.

• Command can be used in Hub chats and private chats
• It only lists one level directory, not recursive
• Only file/folder name, no size or any other meta data
• Usernames will be searched and matched, so you don't have to type the exact username
  • example: real username [prefix]pepsi - /list pepsi /share should find the correct user, unless there is multiple users with names too similar

[Kcchouette]

@peps1 at the origin, the goal was to export file list so in case user is offline we can parse it anyway as it's cached. The second utility I found usefull is sometimes your hard drives died, and if someone has the filelist you can recreate your share by watching which files you have in it, and download it again if your hub has them

[wolfmanx]

The match queue command already saves a DC++ standard file list in config/FileLists. This file list is compatible with other DC++ clients. It is a bztip2(1) compressed XML file. Converting it to other formats is trivial. I have been using these file lists with some python scripts to perform various administrative tasks (e.g. mirroring, without duplicating unshared files).

I could not -- for the life of me -- find a way to trigger the match queue command in the web front end. So, a user command

/match USER

would be the optimal solution to exporting file lists for other users on demand.

As for the own file list. It is already available in the same standard format under files_0_3.xml.bz2 in the config directory. Which makes it a non-issue.

My current workaround to trigger the filelists/match_queue manually is using the web API, e.g.:

# match queue Doombot
curl --include \
     -u someone:important \
     --request POST \
     --header "Content-Type: application/json" \
     --data-binary "{ \
  \"user\": { \
    \"cid\": \"YAQA2K56OTD6SDBAKFRYPGF5XQPZZKWYZFYFEOA\", \
    \"hub_url\": \"one.comicshack.org:411\" \
  }, \
  \"directory\": \"/\" \
}" \
'http://127.0.0.1:5600/api/v1/filelists/match_queue'

[peps1]

@wolfmanx: Thanks! i did not know of either of those. There is currently no trigger like /match USER for the webui, but i'll add it to my todo for the user-commands extension.

Do you know when the local file list files_0_3.xml.bz2 is refreshed/created? It doesn't seem to be on application start and closing / refreshing my own file list didn't cause it to refresh.

[wolfmanx]

@peps1 The algorithm for flushing the own file list to disk has been a long time mystery for me with LinuxDC++ already. There was no discernible connection between refresh runs and file list generation. All I could tell, was that it was synchronized eventually (sometimes a day later!). There was also no effect, when restarting the application.

Looking at the code in airdcpp-webclient/airdcpp-core/airdcpp/ShareProfile.cpp, there seems to be a general 15 minute delay after generationFinished(). Also, file lists cannot be modified, while they are transferred. So it can be tricky ...

[maksis]

The application itself has no need to have an up-to-date compressed filelist to be sitting in the config directory, it's being generated only when it needs to be uploaded to someone....

[wolfmanx]

Not even then. It is sufficient that the complete compressed file resides in memory. However, I would still expect the file list to be saved, when the application exits to have it readily available on next startup. But since the file list can be just as easily generated from the hash database, there is no need for a stored copy at all. But as I said, that's mostly conjecture.

[wolfmanx]

I find it strange, that the application refuses to execute `filelists/match_queue* for the own nickname. That would provide for a simple uniform solution for all file lists. The file list downloading part could easily be executed, while the superfluous match step can be omitted.

[wolfmanx]

And finally, here is some anecdotal evidence for AirDC++. The last hashing operation was reported at 18:08:

[2021-09-26 18:08] Hashing finished: 2 files (134.85 MiB) in 2
directories have been hashed in 2 seconds (59.48 MiB/s)

A new file list was generated 10 minutes later:

  -rw-r--r--  1 ws ws 59619160 Sep 26 18:18 files_0_15.xml.bz2

So, automatic file list storage seems to be close enough.

I found the following comment on a hub, but the command /generatelist does not seem to be doing anything on the web client.

- [2021-09-25 17:45:40] <zzzzz> true, and just before that (is
  you use AirDC) do a /generatelist on the chat bar, this will
  update your file list... in case of HD lost you'll be able to
  load that list and download again whatever you had, maintaining
  your folder classifications and file naming

[peps1]

I see, yes my file list also refreshed some time after hashing.

The WebUI doesn't have the /generatelist command currently, and i don't think there is an API endpoint exposed to trigger it. @maksis what do you think?

I also noticed the files_0_15.xml.bz2 file can't be opened in the Windows client unless it's renamed.