airdcpp-web / airdcpp-release-validator

AirDC++ extension that performs various validations for release directories
1 stars 3 forks source link

Excited... #1

Closed KipourosV closed 4 years ago

KipourosV commented 7 years ago

I have been working with a friend on a couple of scripts for sfv and dupes checking.

Take a look... https://github.com/KipourosV/MusicTools Fairly simple python scripts.

For the SFV check part, my idea goes like this: There are 4 states for the release(folder).

First argument is the root path that will be recursively scanned. When you run the script with no second argument, it just reports the subfolders(releases) state.

Second argument is the state that you want checked. When you run the script with second argument (for example new) the script sfv checks all the folders that are "new", then writes a state file in each checked directory, 0byte size and named after the resulting state.

The second script recursivelly parses the given folder and does a fuzzy search on the folder names in an attempt to locate from exact duplicates to similar release names (ex. from different groups). It also has a similarity ratio that you can tweak.

The idea behind all this is to automate: 1) mass (auto)-redownloading incomplete releases 2) mass deleting garbage folders 3) compare releases from different groups (and keep the best quality rls based on size) 4) (auto)cleaning the share of duplicates 5) being able to initiate a targeted sfv recheck

I would really like to help out with this extension in any way I can. I need some guidance, I m not advanced in programming but I learn quickly.

How do I work with your code here? Where do I need to place what in order to start looking at how the extension works and what it does?

maksis commented 7 years ago

Oh, share scanner in AirDC++ was one of the first real application features that I've ever developed. That feature is old and well tested but it shouldn't really be part of the core application so I'd like to move all the validation functionality into this extension and make things more modular (just look at the actual validation function: https://github.com/airdcpp/airgit/blob/master/airdcpp/airdcpp/modules/ShareScannerManager.cpp#L331-L578).

Auto download of missing files is certainly a popular request. One way to allow that would be to make it possible for extensions to communicate with each others, so that you could have another extension listening for "directory scan failed" events from this extension and performing actions based on them. My plan is to keep this extension focused on the directory content structure validation and let other extensions handle dupe checks, CRC validations and other features (those are quite obvious starting points for extension development as well).

I'll post an update once extension support has been enabled in the develop branch as you can't do much before that.

KipourosV commented 7 years ago

I will do some reading on nodejs as am fairly new to Javascript... Really looking forward to this!

maksis commented 7 years ago

Extensions are now supported in the develop version. Extensions written in other languages are supported as well (I need to document that at some point) but yes, Node will be the primary one as it's going to be shipped with the Windows version.

maksis commented 7 years ago

@KipourosV Do you find the starter project and examples to be useful?

Some feature ideas for this extension:

KipourosV commented 7 years ago

I haven't had time to fiddle with it yet. I'm in the middle of recovering a degraded ceph storage cluster and a pregnant wife.

I'll give you an update this weekend...

maksis commented 6 years ago

Have you abandoned this project? :p

KipourosV commented 6 years ago

No and yes, I've been using the scripts to do the job since I don't have enough time to dive into nodejs.

Any news from your side? Any ready code that I could look at to get an example?

On Mon, Apr 16, 2018 at 6:34 PM maksis notifications@github.com wrote:

Have you abandoned this project? :p

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/maksis/airdcpp-release-validator/issues/1#issuecomment-381667027, or mute the thread https://github.com/notifications/unsubscribe-auth/APmCiORaMCL_u6faUAYEgVOWh5oA9ZnTks5tpMgYgaJpZM4MwMtG .

maksis commented 6 years ago

The starter project might be helpful: https://github.com/airdcpp-web/airdcpp-create-extension

I've also written a really simple example Python extension: https://github.com/airdcpp-web/airdcpp-example-python-extension

KipourosV commented 6 years ago

So here is a nice idea... In some hubs I'm connected, there are commands like: !releases Goa-FLAC that msg you back something like this:


| ●--------------------------------------------------------------HUB NAME-------------------------------------------------------------● |

                                                                                                   ThE   300  NeWeSt  Goa-FLAC  ReLeAsEs

No: 52818 | 04/16/2018 | Goa_Luni-Holographic_World-(TIMEWARP080)-EP-FLAC-2018-LUR | by: [Natal.Bua No: 52421 | 04/02/2018 | Psychedelic_Quest-Out_Of_Time-(TIMEWARP079)-EP-FLAC-2018-LUR | by: [Natal.Bua No: 52174 | 03/19/2018 | TheManiac-_Necronomicon_II-(TIMEWARP078)-WEB-FLAC-2018-VL | by: [[[Pr3.anounce No: 52023 | 03/13/2018 | VA-InnerUniverse-REPACK-WEB-FLAC-2016-LUR | by: [Natal.Bua No: 51998 | 03/12/2018 | Neogenia-_No_LongerHuman-(TIMEWARP077)-WEB-FLAC-2018-VL | by: [[[Pr3.anounce No: 51907 | 03/11/2018 | VA-_MoonRitual-(AMACD002)-CD-FLAC-2014-VL | by: [[[Pr3.anounce No: 51905 | 03/11/2018 | VA-_JellyMarbles(Compiled_By_Filipe_Santos)-(PHTRCD01)-WEB-FLAC-2008-VL | by: [[[Pr3.anounce No: 51897 | 03/11/2018 | Neogenia-No_Longer_Human-(TIMEWARP077)-EP-FLAC-2018-LUR | by: [Natal.Bua No: 51813 | 03/06/2018 | Mind_Echo-Dreams_OfTravels-(TIMEWARP076)-EP-FLAC-2018-LUR | by: [Natal.Bua No: 51359 | 02/24/2018 | VA-_Epoch_Of_TheTerrans-(SUNCDLE04)-CD-FLAC-2014-VL | by: [[[Pr3.anounce No: 51355 | 02/24/2018 | VA-_Future_Architecture2(Compiled_By_Imba_and_Richpa)-(NEOG025)-WEB-FLAC-2014-VL | by: [[[Pr3.anounce No: 51353 | 02/24/2018 | The_Maniac-Necronomicon_II-(TIMEWARP078)-EP-FLAC-2018-LUR | by: [Natal.Bua

It would be nice if we could copy paste that msg to a plugin/addon that would automatically parse the text and auto download all...

KipourosV commented 6 years ago

Also another nice idea would be to have a window of it's own with buttons to initiate the release validator for example, and maybe some sort of activity display (is it running? what is it doing?). Instead of flooding the main log, it could have a window of it's own...

KipourosV commented 6 years ago

Also, the results of the release validator (for example the folders that are missing some files) could be copy pasted to the auto downloader extension that I mentioned earlier. So you can get the files you are missing automatically. Also a way to auto download them at the duplicate location would be sweet!

maksis commented 6 years ago

Also, the results of the release validator (for example the folders that are missing some files) could be copy pasted to the auto downloader extension that I mentioned earlier. So you can get the files you are missing automatically. Also a way to auto download them at the duplicate location would be sweet!

Yes, an extension like that should be doable already (add a setting text area and subscribe for setting updates)

maksis commented 6 years ago

There's also an API method that will return dupe paths in share based on the release name: https://airdcpp.docs.apiary.io/#reference/share/generic-methods/get-dupe-paths

KipourosV commented 6 years ago

Can you show me an example of how you initiate a search through an api call? And also how to initiate a download?

On Tue, Apr 24, 2018 at 10:45 PM maksis notifications@github.com wrote:

There's also an API method that will return dupe paths in share based on the release name: https://airdcpp.docs.apiary.io/#reference/share/generic-methods/get-dupe-paths

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/maksis/airdcpp-release-validator/issues/1#issuecomment-384074058, or mute the thread https://github.com/notifications/unsubscribe-auth/APmCiI3ozWW8s6Kamo_UHdiven_HB_EKks5tr47igaJpZM4MwMtG .

maksis commented 6 years ago

Searching is described (with example code) in the API docs: https://airdcpp.docs.apiary.io/#reference/searching

If you also want to add downloads, that case is included in the JS example scripts: https://github.com/airdcpp-web/airdcpp-extension-js/blob/master/examples/airdcpp-auto-downloader.js

KipourosV commented 6 years ago

I'm gonna start working on this, waiting on a pro progamer friend for some help. I have so many ideas for this... Most hubs have release announcements, on main chat or via an msg. The extension should be able to parse and auto download, according to your filter. For example... This is the announcement:

[[[Pr3.anounce added to the category [Psychedelic-FLAC] the following Release: VA_-Mush-A-Holic(Compiled_By_DJ_Mush)-(WMCD014)-CD-FLAC-2007-VL

So in the extensions settings you can have filters that you can enable/disable/add/remove like :

Filter name: PsyFlac (friendly name) Filter value: [[[Pr3.anounce added to the category [Psychedelic-FLAC] the following Release:

I'm getting to work!

On Sat, Apr 28, 2018 at 12:02 PM maksis notifications@github.com wrote:

Searching is described (with example code) in the API docs: https://airdcpp.docs.apiary.io/#reference/searching

If you also want to add downloads, that case is included in the JS example scripts: https://github.com/airdcpp-web/airdcpp-extension-js/blob/master/examples/airdcpp-auto-downloader.js

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/maksis/airdcpp-release-validator/issues/1#issuecomment-385161987, or mute the thread https://github.com/notifications/unsubscribe-auth/APmCiO7bPLoieJohJJi5y3UyZVvEG7Jnks5ttD4tgaJpZM4MwMtG .

maksis commented 4 years ago

Feel free to open a new issue (or visit the dev hub) if you have additional questions