dlang / dub-registry

Online registry for dub packages
Boost Software License 1.0
82 stars 64 forks source link

Fetching updated packages failed: Missing non-optional field 'score' of type 'DbPackageStats' (DefaultPolicy(T)). #285

Open timotheecour opened 6 years ago

timotheecour commented 6 years ago

@s-ludwig fix for https://github.com/dlang/dub-registry/issues/282#issuecomment-367665542 still fails with: ./dub-registry --mirror=https://code.dlang.org object.Exception@../../../../.dub/packages/vibe-d-0.8.3-alpha.4/vibe-d/mongodb/vibe/db/mongo/collection.d(323): Aggregate command failed: The 'cursor' option is required, except for aggregate with the explain argument

however it makes some progress (finally!) if i replace dub.selections.json with:

"vibe-d": {"path":"../../../D/vibe.d"}

however it gives another error in this case:

./dub-registry --mirror=https://code.dlang.org
[main(----) INF] Listening for requests on http://127.0.0.1:8005/
[main(----) INF] Added virtual host http://code.dlang.org:8005/ (127.0.0.1)
[main(AxpS) INF] Polling 'https://code.dlang.org/' for updates...
Fetching updated packages failed: Missing non-optional field 'score' of type 'DbPackageStats' (DefaultPolicy(T)).
wilzbach commented 6 years ago

This is due to the registry not being updated yet. Two ways to fix this for now:

1) optional

diff --git a/source/dubregistry/dbcontroller.d b/source/dubregistry/dbcontroller.d
index 5c55647..da6eaa6 100644
--- a/source/dubregistry/dbcontroller.d
+++ b/source/dubregistry/dbcontroller.d
@@ -437,7 +437,7 @@ struct DbPackageStats {
    SysTime updatedAt;
    DbDownloadStats downloads;
    DbRepoStats repo;
-   float score = 0; // 0 - invalid, 1-5 - higher means more relevant
+   @optional float score = 0; // 0 - invalid, 1-5 - higher means more relevant
    enum minScore = 0;
    enum maxScore = 5;

2) Set the score on an update

diff --git a/source/dubregistry/mirror.d b/source/dubregistry/mirror.d
index 127c758..7d071a6 100644
--- a/source/dubregistry/mirror.d
+++ b/source/dubregistry/mirror.d
@@ -43,7 +43,17 @@ void mirrorRegistry(DubRegistry registry, URL url)
 nothrow {
    logInfo("Polling '%s' for updates...", url);
    try {
-       auto packs = requestHTTP(url ~ InetPath("api/packages/dump")).readJson().deserializeJson!(DbPackage[]);
+       import std.functional : pipe;
+       import std.algorithm.iteration : each;
+       auto packs = requestHTTP(url ~ InetPath("api/packages/dump"))
+           .readJson()
+           .pipe!((j){
+               j[].each!((p){
+                   p["stats"]["score"] = 0.0;
+               });
+               return j;
+           })
+           .deserializeJson!(DbPackage[]);

        bool[BsonObjectID] current_packs;
        foreach (p; packs) current_packs[p._id] = true;
wilzbach commented 6 years ago

however it makes some progress (finally!) if i replace dub.selections.json with:

dub add-local should work too. Anyhow, let's bump the dependencies:

https://github.com/dlang/dub-registry/pull/288