search

jesus2099 / konami-command

power‐ups for various web sites
123 stars 27 forks source link

Migrate dynamic collection updater to Web Service mode #64

Open jesus2099 opened 9 years ago

jesus2099 commented 9 years ago
jesus2099 commented 8 years ago

This ticket would also fix dynamic updater on 10+ medium releases (#133).

jesus2099 commented 3 years ago

This would also fix (bottom work credits) Highlighting of indirectly related works #612

jesus2099 commented 3 years ago
jesus2099 commented 3 years ago

Work in progress blocked by MBS-11905:

https://github.com/jesus2099/konami-command/commits/ws-dynamic-updater_issue-64 Just one commit at the moment https://github.com/jesus2099/konami-command/commit/1f992228f82739ef33226c33d084f1dde2139614

jesus2099 commented 3 years ago

https://tickets.metabrainz.org/browse/MBS-11905?focusedCommentId=60125&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-60125

Nicolás Tamargo Added 2021-08-23 19:20

The limit is 500 recordings, IIRC, and I don't think there's currently a way around it.

jesus2099 commented 3 years ago

https://github.com/metabrainz/musicbrainz-server/blob/28916986f5f69a3fa9d081a770531f5f6983bc08/lib/DBDefs/Default.pm#L463-L466

# On release browse endpoints in the webservice, we limit the number of
# releases returned such that the total number of tracks doesn't exceed this
# number.
sub WS_TRACK_LIMIT { 500 }
jesus2099 commented 3 years ago

https://github.com/metabrainz/musicbrainz-server/blob/f7d1d109e51b9c0313de3bdd64c1ce0f543e73f1/admin/DumpJSON#L41-L44

=head1 SYNOPSIS
[…]

Releases with more than 500 recordings ignore 'recording-level-rels', and
since those recordings aren't dumped separately, their relationships will
only be included in the target entities' output. As of 2017-04, this affects
~200 releases.

[…]
=cut
jesus2099 commented 3 years ago

https://github.com/metabrainz/musicbrainz-server/blob/f7d1d109e51b9c0313de3bdd64c1ce0f543e73f1/lib/MusicBrainz/Server/Controller/WS/2/Release.pm#L145-L150

        # The maximum number of recordings to try to get relationships for, if
        # inc=recording-level-rels is specified. Certain releases with an enourmous
        # number of recordings (audiobooks) will almost always timeout, driving up
        # server load average for nothing. Ideally we can remove this limit as soon as
        # we resolve the performance issues we have.
        my $max_recording_relationships = 500;
jesus2099 commented 3 years ago

https://github.com/metabrainz/musicbrainz-server/commit/cff992802e1e28e5511e3a2b05fb10974a4bf85b

Limit WS release lists to 500 tracks

This limits the results of release browse queries, including those from the discid endpoint, to contain no more than 500 tracks (however, it always returns at least one release, which may have more than 500 tracks on it).

With this change, certain browse queries with inc=recordings+artist-credits specified can actually have a chance to complete and not time out (which is more helpful to the client, since they'll actually get information back, and more helpful to the server, since it won't increase server load and hog worker processes for no end purpose).

jesus2099 commented 3 years ago

https://github.com/metabrainz/musicbrainz-server/blob/f7d1d109e51b9c0313de3bdd64c1ce0f543e73f1/lib/MusicBrainz/Server/ControllerBase/WS/2.pm#L319-L344

=head2 limit_releases_by_tracks

Truncates a list of releases such that the entire list doesn't contain more
than C<DBDefs::WS_TRACK_LIMIT> tracks in total (but returns at least one
release). The idea is to limit browse queries that contain an excessive number
of tracks when C<inc=recordings> is specified.

Note: This mutates the passed-in array reference C<$releases>.

=cut

sub limit_releases_by_tracks {
    my ($self, $c, $releases) = @_;

    my $track_count = 0;
    my $release_count = 0;

    for my $release (@{$releases}) {
        $c->model('Medium')->load_for_releases($release);
        $track_count += (sum map { $_->track_count } $release->all_mediums) // 0;
        last if $track_count > DBDefs->WS_TRACK_LIMIT && $release_count > 0;
        $release_count++;
    }

    @$releases = @$releases[0 .. ($release_count - 1)];
}
jesus2099 commented 3 years ago

One good thing is that there is a way to detect that I am on a release that has skipped recording-level-rels, and not on a release that simply has no recording level relationships:

relations in release.media[m].tracks[t].recording.relations

In fact it is very good and logical. When it decides to not send relationships, I think it returns same thing it would return when not asked for recording-level-rels.

jesus2099 commented 2 years ago

Oh no, I lost my work in progress file, because of my recent official version (785427386d632fc3b68a940f4a327e9f025a90ea) that triggered auto-update. I should have saved my work in progress in git. 🤦

jesus2099 commented 2 years ago

So MBS-11905 is not really blocking this. I know recording.relations is absent (instead of empty array) when I got no recording-level-rels because of big release.

jesus2099 commented 2 years ago

Browsing web service JSON responses is also a lot of fine-tuning! so I will try my utmost to reuse the existing functions from initial loading #174, rather than keeping a specific dynamic loader loading.

jesus2099 commented 2 years ago

Dynamic remove not being converted to new mode, included release groups are not removed with dynamic remove (OFF or remove from collection).