A even better downloader enablement mechanism

[CXwudi]

Currently, no way for us to config two same downlaoders with different command line for even same pv service.

So in config.downloader, instead of having each pv service listing all available downloaders, lets it be a list of config with name. And the enablement will use the name as the order of downloaders for that pv service.

[vocadb-video-downloader-new] vvd-downloader/src/main/resources/application.yml (Lines 37-65)

  enablement:
    NicoNicoDouga:  # enable downloaders from NicoNicoDouga's available downloaders. e.g. nndownload
    Youtube:
    Bilibili:

  downloader:
    NicoNicoDouga:
      # settings of youtube-dl/ytp-dl for downloading niconico videos
      # settings must make sure nothing blocks from downloading the video (e.g. don't put --version on yt-dlp)
      # do the same for all other downloaders.
      youtube-dl:
        launch-cmd:
        external-args:

      nndownload:
        launch-cmd:
        external-args:

    Youtube:
      youtube-dl:
        launch-cmd:
        external-args:

    Bilibili:
      youtube-dl:
        launch-cmd:
        external-args:

Open in IDE · Open on GitHub

^{Created from JetBrains using CodeStream}

[CXwudi]

The new downloader enablement machenism will delete the enablement section the downloader section will take place of what to enable, it will be like:

 downloader:
    NicoNicoDouga:
      # settings of youtube-dl/ytp-dl for downloading niconico videos
      # settings must make sure nothing blocks from downloading the video (e.g. don't put --version on yt-dlp)
      # do the same for all other downloaders.
      - name: using yt-dlp 
        type: youtube-dl
        launch-cmd: yt-dlp.exe
        external-args: -v, --username, xxx, --password, yyy

      - name: using youtube-dl
        type: youtube-dl
        launch-cmd: youtube-dl.exe
        external-args:  -v, --username, xxx, --password, yyy

      - name: using nndownload
        type: nndownload
        launch-cmd: nndownload.exe
        username:
        password:
        session-cookie:

    Youtube:
      - name: using yt-dlp 
        type: youtube-dl
        launch-cmd: yt-dlp.exe
        external-args: -v, --username, xxx, --password, yyy

    Bilibili:
      - name: using yt-dlp 
        type: youtube-dl
        launch-cmd: yt-dlp.exe
        external-args: -v, --username, xxx, --password, yyy

where the type and name are needed for all downloader configs, and the rest of the config are customized for each downloader

will need to POC how spring boot conversion will works:

• https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config.typesafe-configuration-properties.conversion
• https://www.baeldung.com/configuration-properties-in-spring-boot#3-custom-converter

I am afraid that Spring boot configuration doesn't support conversion of a map of configuration to custom type like https://stackoverflow.com/questions/64228017/converter-using-configurationpropertiesbinding-not-work-for-complex-source-type

Use this way to get all configuration programmatically https://stackoverflow.com/questions/23506471/access-all-environment-properties-as-a-map-or-properties-object

[CXwudi]

Unfortauntely, we decided to not use spring-boot-starter-configuration to handle business logic validation of our various configs. In fact, nor should it be. So the final configuration will be like:

 downloader:
    NicoNicoDouga:
      # settings of youtube-dl/ytp-dl for downloading niconico videos
      # settings must make sure nothing blocks from downloading the video (e.g. don't put --version on yt-dlp)
      # do the same for all other downloaders.
      - name: using yt-dlp 
        type: youtube-dl
        launch-cmd: yt-dlp.exe
        external-args: -v, --username, xxx, --password, yyy

      - name: using youtube-dl
        type: youtube-dl
        launch-cmd: yt-dlp.exe
        external-args: -v, --username, xxx, --password, yyy

      - name: using nndownload
        type: nndownload
        launch-cmd: nndownload.exe
        username:
        password:
        session-cookie:

    Youtube:
      - name: using yt-dlp 
        type: youtube-dl
        options:
        launch-cmd: yt-dlp.exe
        external-args: -v, --username, xxx, --password, yyy

    Bilibili:
      - name: using yt-dlp 
        type: youtube-dl
        launch-cmd: yt-dlp.exe
        external-args: -v, --username, xxx, --password, yyy

So pretty much we are just mapping to Map<PVServiceEnum, List<RawDownloaderConfig>> where RawDownloaderConfig contains everything that any download config could declare (in above lists, we will have name, type, launch-cmd, external-args, username, password, session-cookie where only name and type are not null), because a POC has shown that @ConfigurationPropertiesBinding doesn't work with Collection type, only for String which is useless

And then we will have custom logic and validation to build up all downloaders, and use FailureAnalyzer to provide user-friendly error msg upon downloader config error