findOne not working on collection

[mikkelking]

I am seeing a problem with Files-Collection which doesn't make sense.

When I do a findOne, it doesn't always return a record from the database, even though I know it's there.

In this image collection I am keeping 2 copies of each image, one being a full sized mugshot, and a second avatar image. They share the same metadata._mid, and the filename is either "avatar" or "mugshot".

Everything seems ok until I iterate over the data in the client, putting together a list of portrait images with this code:

        _.each(users,(u)=>{
          _.each(["mugshot","avatar"],(s) => {
            var selector = { 'metadata._mid': u._id, 'filename': s };
            var mug = Mugshots.findOne(selector);
            if (mug) {
              var fname = s+"-"+u._id;
              dev.portraits[fname] = {
                type: s,
                size: mug.length,
                _id: mug._mid,
                fname: fname
              };
            } else
              console.error("Could not find mugshot for ",selector)
          });
        });

users is an array of meteor users. By rights it should never log an error (as all the images are in the database), but in fact 9 of the 18 records are logged as not found (evenly distributed between "avatar" and "mugshot").

I do subscribe to the Mugshot collection.

Thinking that perhaps it was something to do with asking for each image individually, I added a helper to basically do a similar thing as above, except the helper asked for this

Mugshots.find({"metadata._mid": {$in: uids}}).fetch()

But the results were the same, which puzzles me. http Image fetching (defined in the collection common code) works perfectly using a similar selector (combination of metadata._mid and filename)

So my question is, have you seen this kind of behavior before, is it a quirk of gridfs, and is there anything that can be done to fix it?

Here is my data:

meteor:PRIMARY> db.mugshots.files.find()
{ "_id" : ObjectId("1e27e52bfc7f88f7cc541e56"), "length" : 23153, "md5" : "055799bf468b3c7e72f20797eea1eaf1", "uploadDate" : ISODate("2016-12-14T00:03:08.949Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "6FLouMfz2L5pqubK2" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("2b938f5b4bf3956cbbcde6ca"), "length" : 2148, "md5" : "089c1f3f1770930d71336d1b4511df89", "uploadDate" : ISODate("2016-12-14T00:03:09.007Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "6FLouMfz2L5pqubK2" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("7561f260134a07a9eb1fc59e"), "length" : 40963, "md5" : "a846f327d06be459cc3f311092258e26", "uploadDate" : ISODate("2016-12-29T09:06:29.051Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "Ksm7CvrKAMdNGbxuL" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("df8259305875480655f062d4"), "length" : 40963, "md5" : "a846f327d06be459cc3f311092258e26", "uploadDate" : ISODate("2016-12-29T09:06:31.095Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "Ksm7CvrKAMdNGbxuL" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("dad132b48b474d3b52512e7f"), "length" : 2867, "md5" : "ac99a172752c3b63f05928148f53feff", "uploadDate" : ISODate("2016-12-29T09:06:31.846Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "Ksm7CvrKAMdNGbxuL" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("bb5af839513999c898c9fbf7"), "length" : 25606, "md5" : "f83b1687d606995073fabf6f0a5797c5", "uploadDate" : ISODate("2017-01-01T02:30:12.095Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "3gvbDW2zWFKBAtoPK" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("9c17f3e789941d331e4dc067"), "length" : 2561, "md5" : "25feb5c5a81c7db62b1cb05002e516fc", "uploadDate" : ISODate("2017-01-01T02:30:12.183Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "3gvbDW2zWFKBAtoPK" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("5cb452764452000f94c8f2c8"), "length" : 43524, "md5" : "c2eeaf9b317cce2a7795b5bb8a2891ac", "uploadDate" : ISODate("2017-01-02T09:27:21.626Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "d7p6bF6kESGcoGfan" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("bc8b93e3d549617d04dbb42d"), "length" : 2913, "md5" : "690392a60a2cb357974f72181070cf33", "uploadDate" : ISODate("2017-01-02T09:27:21.777Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "d7p6bF6kESGcoGfan" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("f9306f9ed3085a97d71f1dae"), "length" : 22689, "md5" : "81631c53c0268121f96112813d832f68", "uploadDate" : ISODate("2017-01-03T01:39:27.731Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "T88rcnp6wTCnvji2R" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("68be360941138edd6cdb4458"), "length" : 2206, "md5" : "74b25aa0ffdfc89d60f615dcd94c6f44", "uploadDate" : ISODate("2017-01-03T01:39:27.818Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "T88rcnp6wTCnvji2R" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("6d3eac26fb65dc046ed1f20e"), "length" : 36994, "md5" : "8860f55ed2c0bd27336c288321fdacac", "uploadDate" : ISODate("2017-01-03T02:05:50.530Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "2uNahGBrgKNyjnScF" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("1b8480d5e68d271c2ddd8beb"), "length" : 3230, "md5" : "865e67d44135932d9fa33aa51e58653f", "uploadDate" : ISODate("2017-01-03T02:05:50.622Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "2uNahGBrgKNyjnScF" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("a6c725c8dd37b86ecc0d3efd"), "length" : 39972, "md5" : "ab6004187f3a70c9a4cac88dbf490789", "uploadDate" : ISODate("2017-01-03T02:06:11.982Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "PvuLAMc5mvqGTqTrA" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("58a926b44f76a2dcd9fabdc0"), "length" : 3194, "md5" : "ad006b9bf35ecfe8b3adfdb93bcbdc04", "uploadDate" : ISODate("2017-01-03T02:06:12.056Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "PvuLAMc5mvqGTqTrA" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("5b71b1038efd8152090d9fe9"), "length" : 23654, "md5" : "88994db4b63acfb3f7bd0564cfcf22a4", "uploadDate" : ISODate("2017-01-03T02:06:24.739Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "8AJB57opXWDJAa3fs" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("4aab05ce1b894789b9125075"), "length" : 2085, "md5" : "15e36560e421a84c46611fec17ef75a3", "uploadDate" : ISODate("2017-01-03T02:06:24.810Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "8AJB57opXWDJAa3fs" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("d081b3966f1959c28f0bd705"), "length" : 22058, "md5" : "808f2b1bb400a0868d55b697ef40745a", "uploadDate" : ISODate("2017-01-03T02:09:37.763Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "sNpESnqWKzKLjhaJM" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("1ab1d57978b3d2c316228307"), "length" : 2371, "md5" : "018a39ffcdcbcb9f812ad85007ba7cd4", "uploadDate" : ISODate("2017-01-03T02:09:37.839Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "sNpESnqWKzKLjhaJM" }, "aliases" : [ ], "contentType" : "image/jpeg" }
{ "_id" : ObjectId("25d8249b5fa59befbf838e2b"), "length" : 60951, "md5" : "24b8f8bfbb6c1175aef77f64142de024", "uploadDate" : ISODate("2017-01-03T05:27:39.930Z"), "chunkSize" : 2096128, "filename" : "mugshot", "metadata" : { "_mid" : "s9TCBjF6TtrdBR584" }, "aliases" : [ ], "contentType" : "image/jpeg" }
Type "it" for more
meteor:PRIMARY> it
{ "_id" : ObjectId("b62a815f85cf7d3039cc19a5"), "length" : 3579, "md5" : "0395d20e13b28d6d983dce8471471b26", "uploadDate" : ISODate("2017-01-03T05:27:40.043Z"), "chunkSize" : 2096128, "filename" : "avatar", "metadata" : { "_mid" : "s9TCBjF6TtrdBR584" }, "aliases" : [ ], "contentType" : "image/jpeg" }
meteor:PRIMARY>

[vsivsi]

Hi thanks for the super well constructed issue/question!

So, I'm a bit stumped, but have a couple of ideas to try. Firstly, file-collection doesn't implement fc.find() or fc.findOne(). Those fall back to the standard Meteor Mongo.Collection implementations because they don't touch on any gridFS magic at all. You are just asking for one or more of the documents in the [root].files collection.

You say that the HTTP support does work though, which sorta makes sense, because file-collection handles that lookup itself.

Here's a quick test: When you get a lookup failure, try using the same selector with fc.findOneStream() and see if it returns a valid stream. That should be equivalent to the HTTP lookup/response.

Another quick test is to see if file-collection has anything to do with it... You should be able to open the [root].files Collection as a regular Mongo.Collection in Meteor. If so, then your code above should be able to run against that collection in place of the file-collection. If it works fine in that case, then something pretty spooky is going wrong in file-collection. If it fails in the same way you are seeing, then either there is a bug in Meteor itself (seems unlikely) or there is something subtler going on in your code that neither of us realizes yet...

Anyway, as I said, stumped, but if you can try the above tests, it should help narrow it down.

[dpatte]

I'm seeing something similar. The record is in the collection but findOne(new Mongo.ObjectID(x)) doesn't always work. I'm wondering if the resumableIndexName is related to this issue. At some point I changed the index names for some of my FileCollections, and Studio3T sometimes show the indexnames for some collections and not for others.

[dpatte]

In my case, it was an error in a publish statement. I realized the .files were not available on the client, so your suggestions above were useful for me. Thx.