Separate File Repositories

[vickyorlo]

Currently, hydrus has support for only one file repository. This is an issue for users who want to keep multiple types of content in hydrus, but keep them separate. For example, one may want to keep their dank memes but also photos from the summer vacations, but they do not want to see them together pretty much ever, especially when making simple queries like system:inbox.

There are currently "workarounds" - using special marker tags or flat out using a different database, however they are both very inconvenient - the former is simply a lot of work and requires the user to specify which set of media they want to view, and the latter results in duplicated data or work because of synchronized remote tag repositories or other settings.

The proposed solution is to introduce support for multiple local repositories, much like there is currently support for multiple local tag repositories. The user can pick which repositories they want to query, or perhaps multiple. Every local file repository could have its own local tag repository to keep it simple, but perhaps arbitrary tag-file repo connections could also be interesting, depending on how viable they are to implement.

[ShadowJonathan]

Iirc, there were already multiple local repositories implemented, right?

Tags are per hash, so if you'd tag a file on another repo with a tag, it'll stay in that tag repo till you move it, but it'll recognise that file from another repo regardless.

[imtbl]

The correct terminology for this in Hydrus client is service, by the way. Repository refers to services in Hydrus server as seen from the client's perspective (e.g., the PTR).

Iirc, there were already multiple local repositories implemented, right?

Yes, multiple tag services are already a thing, but multiple file services are not. Tags are per hash per tag service, so you can have different tags for a given file in different tag services (and you can limit the search to a specific service).

So the proposal would be to have the same for file services (letting you restrict the search to a specific file service) so you don't have to apply any of the mentioned workarounds.

[bbappserver]

I would argue if you aren't using a different database and/or file storage area they aren't strictly separate. Thus the only way to actually partition hydrus is to use these launch flags or to use two copies of it at different locations.

If you are using the same database, then applying a tag is logically equivalent to them being partitioned. Therefore if you wish to partition files in a folder like way, simply apply an appropriate local tag

[Mengmoshu]

@bbappserver If the original issue hadn't already acknowledged your suggested solutions then your comment might have been productive. All you've done here is prescribe workflows and try to shut down a feature suggestion 8 people support.

[bbappserver]

Ok, so let me lay this out technically, and see if I've understood. There exists

hashes(hash_id,hash)
tags(tag_id,name)
local_mapping(hash_id,tag_id)

You wish to introduce

repo(repo_id,name)
repo_membership(hash_id,repo_id)

You understand these new tables are equivalent to the tag tables all but in name. So is there a reason they need to be a separate feature? Do repos have any special quality other that tags do not, or do you just want certain tags presented differently in the UI?

I can see no reason why there would be a conflict for local tag repo:holiday photos && other search criteria

And this requires no rewriting of search or tagging functionality.

If you don't want to accidentally pull up your racy anime photos or have your exhentai parse start while you're showing off vacation photos at work, then I would still recommend the -d launch option, to fully switch out your tag suggestions, parsers, and downloaders.

[vickyorlo]

Did you even read the OP? I bring up both of the things you're talking about.

[vickyorlo]

To wit - it's true that this could be implemented on the "back end" via the same mechanism as tags. However, the suggestion is more interested in the functionality of it - I don't want to bother with remembering to include some extra tag and exclude all others (if I have some files in multiple file services) every time i make a query, I don't want to bother with keeping two databases when I, for example, want to be synched to the PTR on both.

[bbappserver]

I have read the original post but

1. It is not a workaround to use a local tag technically there is very little difference.
2. I don't understand how introducing another repository is functionally any more convenient for use tagging images.
3. If you are using a local tag it will already not be pushed to the PTR

the former is simply a lot of work and requires the user to specify which set of media they want to view [...] dank memes but also photos from the summer vacations, but they do not want to see them together pretty much ever. [...] they do not want to see them together pretty much ever, especially when making simple queries like system:inbox.

Is not the intention of multiple repos to do exactly this with just a slightly different UI that uses a list of repo names instead of a tag? That is how does a repo differ from the following local tag query (is it just a different GUI with like a combobox, and how much faster is this really than typing a string?) repo:meme && system:inbox &&... repo:personal && system:inbox &&...

If you want a special additional feature does it overlap in any way with groups?https://github.com/hydrusnetwork/hydrus/issues/310

Every local file repository could have its own local tag repository to keep it simple, but perhaps arbitrary tag-file repo connections could also be interesting, depending on how viable they are to implement.

What workflow precisely is this simplifying? And if you wanted to prevent data duplication with multiple databases,then how does duplicating definitions help? If you want arbitrary tag-file repo connections then why did you want to partition the repos in the first place?

It just seems like a lot of work to change something that hydrus already does pretty well, and is introducing a whole table and join on every query so I can't see an efficiency argument for it, unless all you really want is for hydrus to have a preset list of filter tags. But maybe there is a very specific benefit I'm not seeing of making a clean separation, or is somehow not covered by the above?

What I don't understand is the want to introduce a special class of label which is as far as I see based on the OP functional equivalent to a tag. But I could see that maybe you want to introduce a small GUI change to quickly filter (possibly an OR filter for "searching a subset of repos") some favourite tags.

[bbappserver]

To wit - it's true that this could be implemented on the "back end" via the same mechanism as tags. However, the suggestion is more interested in the functionality of it - I don't want to bother with remembering to include some extra tag and exclude all others (if I have some files in multiple file services) every time i make a query, I don't want to bother with keeping two databases when I, for example, want to be synched to the PTR on both.

That being the case could you specify clearly:

• In a way which is testable how you would like the GUI modified to accommodate, and
• do you care that the filtering label would reside in the same tablespace as existing tags?

e.g.

• The tags management dialog shall be extended to show a seperate tab for each of my repositories
• Tag repositories shall be managed in preferences as a strings list under the "tagging" item
• The repositories to search may be specified either as:
  • A checked box in the search pane in a combobox dropdown
  • A tag in the repo: namespace
• The resultant query shall display as an OR search of repo tags
• A file repository must not be confused with a tag repository they are distinct.
• A file repository stores a collection of file, but may use any tag from the combined virtual tag service.
• A file repository may at the discretion of the implementer specify additional tags not in the local or PTR tag services.
• A downloader may add a file to a file repository by using its corresponding repo tag name in the tag import dialog used in gallery downloader pages and subscriptions.
• A tag repository name may have an uppercase displayname, but must have an equivalent lowercase name for the purposes of searching and tagging by downloaders, if multiple repository names result in the same lowercase tag name, hydrus must produce an error informing the user ofthe conflict.

[thatfuckingbird]

@bbappserver This is a perfectly valid feature request as it is written, it even used to be on hydev's "big job" list and on the feature votes. Multiple (remote) file repositories already exist, so it is clear how the UI should look like, since it is already mostly there. Currently you just can't have multiple local ones. It is not the same as using repo: tags, because:

• Those would be affected by all mass operations (potentially done through the API) that work on tags. Since there is no way to mark tags as special or attach more metadata to them, it is too easy for some unaware application or feature to mess these tags up.
• Having multiple file repos would allow related features like splitting storage locations based on file repos.
• There is no way to enforce that all files have a repo tag or that they have only one. It is a pain always adding repo: tags in queries.

Downloaders could work the same way as they do now for tags: the file import options could be extended to look like the tag repo selection part of the tag import options.

As for the last paragraph of OP, afaik there is no connection between file and tag repos now, those operate independently.

[hydrusnetwork]

Hey, here is my current list of jobs to do for this project. The dream is this is all done by Q3, but I expect there will have to be some compromises.

• [x] cleaning up legacy hardcoded 'my files' presentation
• [x] updating deleted tracking to hold timestamps
• [x] updating trash to hold timestamps and deal with multiple services
• [x] wrote a new object to handle search contexts for multiple local file services
• [x] all file search, and UI, uses the new location search context
• [x] write a widget to allow multiple file service selection for the new location context and plug it into autocomplete dropdown
• [x] cleaning up legacy hardcoded 'my files' selection, and adding 'file import options' stubs for import destination
• [x] cleaning up legacy hardcoded content update code, figuring out multiservice trash and undelete
• [x] update existing client api calls to deal with location search contexts
• [x] cleaning up legacy hardcoded 'my files' unit tests
• [x] convert last remaining references to 'my files' to newer 'default my files' system, like with 'my tags'
• [x] write new database content updates for file move and copy, and menu actions for it
• [x] and unit tests
• [x] update client api to do these actions, and file delete
• [x] write file delete/merge/whatever routines on how to deal with deleting a local file service
• [x] enable multiple local file services in manage services
• [x] brush up system predicates for file location searching

[CuddleBear92]

closing this as i believe its done for a while now.

[Zweibach]

Read the damn todo-list.

[hydrusnetwork]

Finally ticking off the last Client API work today with 586 coming out tomorrow. I added multiple file domain destination support for add file and add URL, and added the ability to copy files from one local file service to another.