search

pgvector / pgvector-python

pgvector support for Python
MIT License
962 stars 62 forks source link

Advice for properly timing Django queryset evaluation #61

Closed rsomani95 closed 8 months ago

rsomani95 commented 8 months ago

Hello, I was hoping to get some advice on how to best go about scoping Django querysets that hold vector data to ensure that the CosineDistance calculation is properly scoped to the relevant rows only. Here's how I'm currently querying the Django ORM:

Python:

from pgvector.django import CosineDistance

user_id = '...'
collection_id = '...'
distance = 0.8

scoped_queryset = (
    VSFItem.objects
    .filter(seg__assets__user__id=user_id)
    .filter(seg__assets__collection__id=collection_id)
)
embedding = VSFItem.objects.first().vector
similar_faces_qset = (
    scoped_queryset
    .alias(distance=CosineDistance("vector", embedding))
    .filter(distance__lt=distance)
)

Which, on calling print(similar_faces_qset.query), shows that the following SQL code has been generated:

SELECT "vsf_item"."seg_id",
       "vsf_item"."vector",
FROM   "vsf_item"
       INNER JOIN "segs"
               ON ( "vsf_item"."seg_id" =
                    "segs"."id" )
       INNER JOIN "assets"
               ON ( "segs"."asset_id" =
                    "assets"."id" )
WHERE  ( "assets"."user_id" =
         b679c79c - 0684 - 4f92 - 9a35 - 29824d62e569
         AND "assets"."collection_id" =
             e3faf9d4 - a7dc - 4710 - ad27 - 85619d34619c
         AND ( "vsf_item"."vector" <=> [-0.734, 0.1616 ... ]) < 0.8 )

The WHERE clause here, specifically the AND ( "vsf_item"."vector" <=> [-0.734, 0.1616 ... ]) < 0.8 ) suggests to me that the vector calculation might be happening across all rows of the "vsf_item" table. Is that accurate? If so, do you have any recommendations on how to best scope computation given the above queries? (I'm doing scoping similar in spirit to the above in different parts of my project, so was hoping for general guidelines re. best practices).

Thanks!

Adjacent issue that's worth mentioning: https://github.com/pgvector/pgvector/issues/203#issuecomment-1657258655

ankane commented 8 months ago

Hi @rsomani95, Postgres will figure out the best way to execute the query (in most situations), which involves minimizing work. You can use EXPLAIN ANALYZE to see the details.

rsomani95 commented 6 months ago

@ankane EXPLAIN ANALYZE was helpful, thanks. It suggests to me that the concern presented above is true i.e. the vector filter is being applied on all rows of the table.

First, here's a simplified version of the code above:

        # `embeddable_cls` is a Django Model
        # `candidate_ids` is a list of ids to limit the scope of the vector filtering

        scoped_qset = embeddable_cls.objects.filter(id__in=candidate_ids)

        queryset = (
            scoped_qset
            .annotate(distance=CosineDistance(embedding_name, embedding))
            .filter(distance__lt=query_arguments.distance)
        )

        filtered_ids = list(queryset.values_list("id", flat=True))

In this case, candidate_ids is a precomputed list of 33 ids. The entire table has 178 rows in total. I'd expect the cosine distance to only be computed on those 33 ids, and not 178 rows. However, this is the generated SQL:

Executed SQL
SELECT "ozu_api_segment"."id"
  FROM "ozu_api_segment"
 WHERE ("ozu_api_segment"."id" IN ('36d3abe1-2cf5-4796-8445-a7529ff0c8fb'::uuid, 'f1d02d3d-7861-40c6-95ae-627e791983cf'::uuid, '32a6be12-617b-4ef8-b580-698b99628b01'::uuid, '2f7cb18d-7e09-4b6b-976a-b16db475ce2f'::uuid, '5655dd82-fe55-426d-b54e-38663e1f13a2'::uuid, '1cca49ea-e27f-40af-82f3-d37417073ffe'::uuid, '887ad098-be19-4575-ac30-679cb65a150a'::uuid, 'a48b75f2-ca12-4e52-b843-664067bf03e4'::uuid, '4c72595c-fade-4af2-84e1-723b0dc23422'::uuid, '9973b052-27d0-4378-b7ba-0c56027e3c21'::uuid, '93f1b0aa-ea42-4b85-b216-7bc2b2e15a16'::uuid, 'bb6eded5-8855-4766-b892-dcb420710ec3'::uuid, 'ea2d008f-3108-4fe1-b82b-c0c0b4784005'::uuid, '6e2609a4-4e7a-4fb1-865e-9b9d707b08a1'::uuid, 'a176518f-f7cf-4171-9802-9f191caa7d87'::uuid, '02b0ced8-490b-4ef2-9300-bdf702102843'::uuid, '62e8f839-3023-4da8-b531-59a6dd61fd25'::uuid, 'f74db32a-705f-436b-bcf0-da480e535755'::uuid, 'b8cc4c88-5f4c-4408-aea3-6bac56cd4caf'::uuid, '2825b039-2a51-45cc-83e0-51ad0c4a4af9'::uuid, '33c0bc96-57f6-4415-9b4c-38890c53332e'::uuid, 'd7b35172-d824-43ae-8a89-e1974db7d9f2'::uuid, '7b3ae0c2-6526-437a-b10e-491826d197bf'::uuid, '24a5182d-ce1c-43a8-9dc0-02b0f6fc81a0'::uuid, 'fc94f78c-6120-4efc-8512-5b89f8b85298'::uuid, '2d98b2e0-3c5f-4067-9aea-3f00472342c2'::uuid, 'eda81005-1182-40fd-b486-7d1ea9a690b7'::uuid, '62dedc3d-b2f2-414b-b2bc-6c6bf57b7cda'::uuid, '432b3a21-acc8-416f-921e-541a03eb0460'::uuid, '0ea707b5-a54a-4a59-b362-95b0383c77a5'::uuid, 'cbf00545-2ab4-48ac-bcde-3b535e36e45d'::uuid, 'cbf8e005-5197-441a-ba45-5bb450b69707'::uuid, '6ee8ee78-c360-48c2-b6f5-1bf73cddaa9d'::uuid) AND ("ozu_api_segment"."visual_cinema_clip_embedding" <=> '[0.0003764629364013672,-0.020965576171875,0.04302978515625,0.0278472900390625,-0.0251007080078125,-0.003215789794921875,-0.0175018310546875,0.06768798828125,-0.00933074951171875,-0.007747650146484375,-0.034759521484375,-0.022918701171875,0.0253753662109375,0.0235748291015625,0.043487548828125,0.01320648193359375,0.00804901123046875,-0.004238128662109375,-0.0309906005859375,-0.041534423828125,-0.0245208740234375,-0.0198516845703125,-0.031005859375,-0.061187744140625,0.0034580230712890625,0.01380157470703125,0.00882720947265625,0.0016460418701171875,-0.04913330078125,0.08111572265625,0.08233642578125,0.0239105224609375,-0.0213470458984375,0.066162109375,-0.0027675628662109375,-0.0251617431640625,0.0285491943359375,-0.0015277862548828125,0.0206146240234375,0.0693359375,0.026153564453125,-0.0021762847900390625,0.034698486328125,-0.00156402587890625,-0.020599365234375,0.060394287109375,-0.053131103515625,-0.0164794921875,0.03680419921875,-0.0090789794921875,0.033233642578125,0.0079803466796875,-0.01136016845703125,-0.00011813640594482422,0.09735107421875,-0.004077911376953125,-0.036041259765625,0.01226043701171875,0.0308380126953125,0.00339508056640625,-0.03497314453125,0.02703857421875,-0.028045654296875,0.02752685546875,-0.005718231201171875,-0.0013713836669921875,0.004856109619140625,-0.01421356201171875,-0.0207061767578125,0.052337646484375,-0.004245758056640625,0.04864501953125,0.0146484375,0.05718994140625,-0.01105499267578125,0.020660400390625,-0.0141754150390625,0.048248291015625,-0.044036865234375,0.0196685791015625,-0.0120391845703125,0.0082550048828125,0.01031494140625,0.020599365234375,0.0198211669921875,0.021331787109375,-0.0211029052734375,0.027679443359375,-0.0019426345825195312,0.0173187255859375,0.004001617431640625,-0.068603515625,-0.00823211669921875,-0.00957489013671875,0.059722900390625,0.007549285888671875,-0.0120391845703125,0.086181640625,-0.03289794921875,0.010986328125,-0.021697998046875,-0.084228515625,-0.05377197265625,0.081787109375,0.01377105712890625,0.01006317138671875,-0.044403076171875,-0.045379638671875,0.029083251953125,-0.055206298828125,-0.0205078125,-0.046600341796875,0.065673828125,0.08233642578125,0.0302276611328125,-0.01033782958984375,0.007793426513671875,0.06170654296875,-0.01270294189453125,-0.01256561279296875,0.07794189453125,-0.010528564453125,-0.01049041748046875,0.025848388671875,-0.087158203125,-0.00937652587890625,-0.03289794921875,0.0038127899169921875,0.07275390625,0.054290771484375,-0.026885986328125,-0.00829315185546875,-0.042633056640625,0.0025577545166015625,-0.0017404556274414062,0.0302886962890625,-0.0264434814453125,0.01045989990234375,-0.039154052734375,0.0299530029296875,-0.0036773681640625,-0.071533203125,0.0008149147033691406,0.063720703125,-0.042327880859375,0.01448822021484375,-0.0246734619140625,-0.06878662109375,-0.002063751220703125,0.0078582763671875,0.0012798309326171875,0.01201629638671875,-0.01220703125,0.0279541015625,0.06134033203125,0.07977294921875,-0.0794677734375,-0.004276275634765625,-0.01059722900390625,-0.0411376953125,0.03948974609375,-0.0259246826171875,0.09283447265625,-0.067626953125,-0.052490234375,0.01441192626953125,-0.03497314453125,-0.048858642578125,-0.11151123046875,-0.0335693359375,0.0606689453125,0.0148468017578125,-0.00479888916015625,0.0007290840148925781,0.034027099609375,0.023834228515625,-0.01861572265625,-0.00646209716796875,0.0027523040771484375,0.032073974609375,-0.09417724609375,-0.0217742919921875,-0.0283660888671875,-0.01605224609375,0.0113983154296875,0.1824951171875,0.0107879638671875,-0.019866943359375,0.027679443359375,-0.0411376953125,0.005558013916015625,-0.048614501953125,-0.0231781005859375,-0.00989532470703125,-0.0123138427734375,0.0149383544921875,0.0266571044921875,0.024078369140625,0.0005826950073242188,-0.06280517578125,0.0181121826171875,-0.002910614013671875,-0.0167083740234375,-0.024993896484375,-0.0137939453125,-0.0289154052734375,0.047119140625,-0.025054931640625,-0.006130218505859375,0.0233306884765625,-0.0297698974609375,-0.03363037109375,0.0004024505615234375,-0.0301971435546875,-0.025360107421875,-0.033111572265625,-0.05364990234375,-0.03509521484375,0.0079345703125,0.0298004150390625,-0.004131317138671875,0.031341552734375,0.0289764404296875,0.026702880859375,0.05780029296875,-0.024078369140625,0.005283355712890625,-0.066162109375,0.04840087890625,0.078369140625,0.01142120361328125,-0.004451751708984375,0.044189453125,0.0200347900390625,0.006839752197265625,0.0247955322265625,0.014617919921875,0.003009796142578125,-0.0012531280517578125,0.0194244384765625,-0.034332275390625,-0.06524658203125,0.05279541015625,0.033721923828125,0.039886474609375,0.006412506103515625,-0.0239410400390625,-0.012237548828125,0.050018310546875,-0.057861328125,0.047882080078125,-0.0260772705078125,-0.05377197265625,-0.046966552734375,-0.0280303955078125,-0.041778564453125,0.0249786376953125,-0.005756378173828125,-0.01372528076171875,-0.09185791015625,0.02880859375,0.0198516845703125,-0.0295562744140625,0.0190277099609375,-0.0360107421875,0.055267333984375,-0.0477294921875,-0.076171875,-0.024200439453125,0.035430908203125,0.00417327880859375,-0.0035495758056640625,-0.0599365234375,-0.035400390625,0.004192352294921875,0.0244598388671875,0.026702880859375,0.02545166015625,0.00826263427734375,0.015869140625,-0.075927734375,0.07330322265625,-0.02362060546875,0.01403045654296875,-0.03326416015625,-0.027984619140625,0.06036376953125,0.026092529296875,0.07147216796875,-0.02117919921875,0.043975830078125,0.0131378173828125,0.054168701171875,-0.041259765625,-0.0126953125,0.040740966796875,0.05999755859375,-0.01241302490234375,-0.041534423828125,-0.04473876953125,0.0020465850830078125,-0.0258636474609375,-0.001049041748046875,-0.0169219970703125,-0.03759765625,0.029266357421875,-0.024261474609375,0.036529541015625,0.041168212890625,-0.038482666015625,-0.0032596588134765625,0.07305908203125,-0.05352783203125,0.055755615234375,0.0156707763671875,-0.009429931640625,0.032684326171875,-0.025177001953125,0.01256561279296875,0.0902099609375,0.0992431640625,-0.0174713134765625,-0.00716400146484375,-0.00629425048828125,-0.0204010009765625,-0.090087890625,-0.02880859375,0.01800537109375,-0.030303955078125,0.040496826171875,0.004913330078125,-0.04498291015625,-0.005641937255859375,0.0216064453125,0.11688232421875,-0.0084686279296875,-0.09259033203125,-0.046966552734375,-0.00827789306640625,0.056365966796875,0.0379638671875,-0.010162353515625,-0.056610107421875,-0.006305694580078125,-0.0167694091796875,0.0731201171875,-0.03436279296875,0.04302978515625,0.05078125,0.00962066650390625,-0.01922607421875,0.0225830078125,0.0167388916015625,-0.002910614013671875,-0.042266845703125,0.01123046875,0.0290374755859375,-0.01137542724609375,-0.023529052734375,0.0821533203125,-0.04290771484375,0.08074951171875,0.0003943443298339844,0.0218353271484375,0.0165557861328125,-0.038848876953125,-0.0240478515625,0.062744140625,0.01499176025390625,-0.0030841827392578125,0.03338623046875,0.0237274169921875,-0.07220458984375,0.0221099853515625,0.004241943359375,-0.019866943359375,0.018890380859375,0.0120391845703125,0.0255584716796875,-0.02716064453125,0.0234527587890625,0.09869384765625,0.0156707763671875,0.0018520355224609375,-0.04486083984375,-0.0009112358093261719,-0.023895263671875,0.006744384765625,0.0421142578125,0.0009751319885253906,-0.00916290283203125,-0.0179443359375,-0.015655517578125,-0.004756927490234375,0.02947998046875,-0.0157012939453125,-0.033843994140625,-0.06597900390625,-0.017059326171875,0.03326416015625,-0.03436279296875,0.004993438720703125,0.004909515380859375,-0.0121612548828125,-0.0166015625,0.03436279296875,0.08599853515625,-0.01239013671875,0.045654296875,0.0182647705078125,-0.0145263671875,-0.01454925537109375,0.12371826171875,-0.0200653076171875,0.0037517547607421875,-0.046173095703125,-0.053009033203125,0.01522064208984375,0.07281494140625,-0.0046539306640625,0.026123046875,-0.0257415771484375,-0.0174407958984375,0.02496337890625,-0.08990478515625,-0.02398681640625,-0.07366943359375,-0.0006928443908691406,-0.026123046875,-0.0467529296875,-0.00948333740234375,-0.00968170166015625,-0.005199432373046875,-0.006343841552734375,0.0665283203125,-0.047149658203125,-0.037384033203125,-0.095703125,0.0298004150390625,0.01137542724609375,-0.051300048828125,-0.018585205078125,0.00933074951171875,0.026641845703125,-0.0272064208984375,-0.0601806640625,-0.016265869140625,0.0298309326171875,-0.007411956787109375,0.03857421875,0.04473876953125,0.046142578125,-0.0122833251953125,-0.04132080078125,0.0947265625,-0.04644775390625,0.0139312744140625,0.051971435546875,-0.00021028518676757812,-0.00450897216796875,-0.07122802734375,-0.0167388916015625,0.06585693359375,-0.0048370361328125,-0.0232391357421875,-0.033050537109375,0.0260162353515625,0.0013494491577148438,-0.006282806396484375,-0.03387451171875,-0.0033740997314453125,0.0009613037109375,-0.02777099609375,-0.003681182861328125,0.0229034423828125,-0.0035457611083984375,-0.027191162109375,-0.001247406005859375,0.01039886474609375,0.0186767578125,0.041778564453125,0.0207366943359375,0.12469482421875,0.0084991455078125,-0.061187744140625,-0.008087158203125,-0.0009169578552246094,-0.054534912109375,0.00316619873046875,-0.051849365234375,-0.04547119140625,0.046051025390625,-0.0165557861328125,-0.027923583984375,-0.01178741455078125,0.0281219482421875,-0.01092529296875,0.052825927734375,-0.0401611328125,-0.018280029296875,0.0447998046875,0.0341796875,-0.01271820068359375,0.041595458984375,-0.00449371337890625,0.027801513671875,-0.03668212890625,0.038818359375,0.05303955078125,-0.028289794921875,0.0479736328125,-0.0192108154296875,0.01331329345703125,0.01372528076171875,0.029327392578125,-0.033111572265625,-0.00835418701171875,0.0168304443359375,0.003765106201171875,0.048980712890625,0.041778564453125,0.00949859619140625,-0.08612060546875,-0.034210205078125,-0.06573486328125,-0.0340576171875,-0.0216064453125,0.019287109375,-0.112060546875,-0.0200042724609375,-0.01360321044921875,0.00970458984375,0.0035800933837890625,0.062744140625,-0.0152435302734375,-0.024871826171875,-0.0330810546875,0.004749298095703125,-0.0294189453125,-0.07647705078125,-0.0202789306640625,-0.061492919921875,0.01383209228515625,0.050140380859375,-0.0513916015625,-0.05206298828125,-0.042694091796875,0.0626220703125,0.0283660888671875,0.0694580078125,0.026458740234375,0.000640869140625,-0.054046630859375,-0.03558349609375,-0.0202789306640625,0.00527191162109375,-0.04998779296875,0.0738525390625,-0.047027587890625,-0.043792724609375,0.0302276611328125,0.0122528076171875,-0.0034275054931640625,0.0360107421875,0.033111572265625,-0.021270751953125,-0.0213775634765625,0.090576171875,-0.032257080078125,0.029296875,0.01522064208984375,0.04644775390625,0.021148681640625,0.061676025390625,-0.11041259765625,0.005359649658203125,-0.03363037109375,0.0218048095703125,-0.01169586181640625,-0.056243896484375,-0.0777587890625,-0.046783447265625,-0.0178070068359375,-0.0109405517578125,0.0015020370483398438,0.055267333984375,0.0008668899536132812,-0.006008148193359375,-0.030364990234375,-0.0142364501953125,-0.0560302734375,-0.011627197265625,-0.038238525390625,-0.0180816650390625,0.037261962890625,-0.0018625259399414062,-0.09649658203125,0.08349609375,-0.04248046875,-0.0031414031982421875,-0.024658203125,-0.045867919921875,-0.005954742431640625,-0.0204925537109375,-0.042877197265625,-0.0081024169921875,0.0272216796875,-0.0166473388671875,0.00867462158203125,-0.05712890625,0.035247802734375,-0.0174407958984375,0.051361083984375,0.043121337890625,-0.013397216796875,0.007686614990234375,0.05181884765625,0.0316162109375,-0.029266357421875,-0.0010929107666015625,-0.014404296875,0.0391845703125,0.0204620361328125,-0.039947509765625,0.0187835693359375,-0.0265045166015625,-0.0158233642578125,-0.0104522705078125,-0.00302886962890625,-0.047149658203125,-0.01099395751953125,0.036529541015625,-0.0012578964233398438,0.08001708984375,-0.01531982421875,-0.03363037109375,0.015960693359375,-0.022125244140625,-0.00878143310546875,-0.0164337158203125]') < 0.6)

and this is the output of EXPLAIN ANALYZE:


QUERY PLAN
Seq Scan on ozu_api_segment  (cost=0.08..9.12 rows=11 width=16) (actual time=0.497..0.638 rows=10 loops=1)
  Filter: ((id = ANY ('{36d3abe1-2cf5-4796-8445-a7529ff0c8fb,f1d02d3d-7861-40c6-95ae-627e791983cf,32a6be12-617b-4ef8-b580-698b99628b01,2f7cb18d-7e09-4b6b-976a-b16db475ce2f,5655dd82-fe55-426d-b54e-38663e1f13a2,1cca49ea-e27f-40af-82f3-d37417073ffe,887ad098-be19-4575-ac30-679cb65a150a,a48b75f2-ca12-4e52-b843-664067bf03e4,4c72595c-fade-4af2-84e1-723b0dc23422,9973b052-27d0-4378-b7ba-0c56027e3c21,93f1b0aa-ea42-4b85-b216-7bc2b2e15a16,bb6eded5-8855-4766-b892-dcb420710ec3,ea2d008f-3108-4fe1-b82b-c0c0b4784005,6e2609a4-4e7a-4fb1-865e-9b9d707b08a1,a176518f-f7cf-4171-9802-9f191caa7d87,02b0ced8-490b-4ef2-9300-bdf702102843,62e8f839-3023-4da8-b531-59a6dd61fd25,f74db32a-705f-436b-bcf0-da480e535755,b8cc4c88-5f4c-4408-aea3-6bac56cd4caf,2825b039-2a51-45cc-83e0-51ad0c4a4af9,33c0bc96-57f6-4415-9b4c-38890c53332e,d7b35172-d824-43ae-8a89-e1974db7d9f2,7b3ae0c2-6526-437a-b10e-491826d197bf,24a5182d-ce1c-43a8-9dc0-02b0f6fc81a0,fc94f78c-6120-4efc-8512-5b89f8b85298,2d98b2e0-3c5f-4067-9aea-3f00472342c2,eda81005-1182-40fd-b486-7d1ea9a690b7,62dedc3d-b2f2-414b-b2bc-6c6bf57b7cda,432b3a21-acc8-416f-921e-541a03eb0460,0ea707b5-a54a-4a59-b362-95b0383c77a5,cbf00545-2ab4-48ac-bcde-3b535e36e45d,cbf8e005-5197-441a-ba45-5bb450b69707,6ee8ee78-c360-48c2-b6f5-1bf73cddaa9d}'::uuid[])) AND ((visual_cinema_clip_embedding <=> '[0.00037646294,-0.020965576,0.043029785,0.02784729,-0.025100708,-0.0032157898,-0.017501831,0.06768799,-0.0093307495,-0.00774765,-0.03475952,-0.022918701,0.025375366,0.02357483,0.04348755,0.013206482,0.008049011,-0.0042381287,-0.0309906,-0.041534424,-0.024520874,-0.019851685,-0.03100586,-0.061187744,0.003458023,0.013801575,0.0088272095,0.0016460419,-0.0491333,0.08111572,0.082336426,0.023910522,-0.021347046,0.06616211,-0.0027675629,-0.025161743,0.028549194,-0.0015277863,0.020614624,0.06933594,0.026153564,-0.0021762848,0.034698486,-0.0015640259,-0.020599365,0.060394287,-0.053131104,-0.016479492,0.0368042,-0.0090789795,0.033233643,0.007980347,-0.011360168,-0.000118136406,0.097351074,-0.0040779114,-0.03604126,0.012260437,0.030838013,0.0033950806,-0.034973145,0.027038574,-0.028045654,0.027526855,-0.005718231,-0.0013713837,0.0048561096,-0.014213562,-0.020706177,0.052337646,-0.004245758,0.04864502,0.0146484375,0.05718994,-0.011054993,0.0206604,-0.014175415,0.04824829,-0.044036865,0.01966858,-0.012039185,0.008255005,0.010314941,0.020599365,0.019821167,0.021331787,-0.021102905,0.027679443,-0.0019426346,0.017318726,0.0040016174,-0.068603516,-0.008232117,-0.00957489,0.0597229,0.007549286,-0.012039185,0.08618164,-0.03289795,0.010986328,-0.021697998,-0.084228516,-0.053771973,0.08178711,0.013771057,0.010063171,-0.044403076,-0.04537964,0.029083252,-0.0552063,-0.020507812,-0.04660034,0.06567383,0.082336426,0.030227661,-0.01033783,0.0077934265,0.061706543,-0.012702942,-0.012565613,0.077941895,-0.010528564,-0.0104904175,0.025848389,-0.0871582,-0.009376526,-0.03289795,0.00381279,0.072753906,0.05429077,-0.026885986,-0.008293152,-0.042633057,0.0025577545,-0.0017404556,0.030288696,-0.026443481,0.0104599,-0.039154053,0.029953003,-0.0036773682,-0.0715332,0.0008149147,0.0637207,-0.04232788,0.01448822,-0.024673462,-0.06878662,-0.0020637512,0.007858276,0.0012798309,0.012016296,-0.012207031,0.027954102,0.061340332,0.07977295,-0.07946777,-0.0042762756,-0.010597229,-0.041137695,0.039489746,-0.025924683,0.09283447,-0.06762695,-0.052490234,0.014411926,-0.034973145,-0.048858643,-0.11151123,-0.033569336,0.060668945,0.014846802,-0.004798889,0.000729084,0.0340271,0.023834229,-0.018615723,-0.006462097,0.002752304,0.032073975,-0.094177246,-0.021774292,-0.028366089,-0.016052246,0.011398315,0.18249512,0.010787964,-0.019866943,0.027679443,-0.041137695,0.005558014,-0.048614502,-0.0231781,-0.009895325,-0.012313843,0.0149383545,0.026657104,0.02407837,0.000582695,-0.062805176,0.018112183,-0.002910614,-0.016708374,-0.024993896,-0.013793945,-0.028915405,0.04711914,-0.025054932,-0.0061302185,0.023330688,-0.029769897,-0.03363037,0.00040245056,-0.030197144,-0.025360107,-0.033111572,-0.053649902,-0.035095215,0.00793457,0.029800415,-0.004131317,0.031341553,0.02897644,0.02670288,0.057800293,-0.02407837,0.0052833557,-0.06616211,0.04840088,0.07836914,0.011421204,-0.0044517517,0.044189453,0.02003479,0.006839752,0.024795532,0.01461792,0.0030097961,-0.001253128,0.019424438,-0.034332275,-0.06524658,0.05279541,0.033721924,0.039886475,0.006412506,-0.02394104,-0.012237549,0.05001831,-0.057861328,0.04788208,-0.02607727,-0.053771973,-0.046966553,-0.028030396,-0.041778564,0.024978638,-0.005756378,-0.013725281,-0.09185791,0.028808594,0.019851685,-0.029556274,0.01902771,-0.036010742,0.055267334,-0.047729492,-0.076171875,-0.02420044,0.03543091,0.004173279,-0.0035495758,-0.059936523,-0.03540039,0.0041923523,0.024459839,0.02670288,0.02545166,0.008262634,0.01586914,-0.075927734,0.07330322,-0.023620605,0.014030457,-0.03326416,-0.02798462,0.06036377,0.02609253,0.07147217,-0.0211792,0.04397583,0.013137817,0.0541687,-0.041259766,-0.0126953125,0.040740967,0.05999756,-0.012413025,-0.041534424,-0.04473877,0.002046585,-0.025863647,-0.0010490417,-0.016921997,-0.037597656,0.029266357,-0.024261475,0.03652954,0.041168213,-0.038482666,-0.0032596588,0.07305908,-0.053527832,0.055755615,0.015670776,-0.009429932,0.032684326,-0.025177002,0.012565613,0.09020996,0.099243164,-0.017471313,-0.0071640015,-0.0062942505,-0.020401001,-0.09008789,-0.028808594,0.018005371,-0.030303955,0.040496826,0.00491333,-0.04498291,-0.0056419373,0.021606445,0.116882324,-0.008468628,-0.09259033,-0.046966553,-0.008277893,0.056365967,0.037963867,-0.0101623535,-0.056610107,-0.0063056946,-0.01676941,0.07312012,-0.034362793,0.043029785,0.05078125,0.0096206665,-0.019226074,0.022583008,0.016738892,-0.002910614,-0.042266846,0.011230469,0.029037476,-0.011375427,-0.023529053,0.08215332,-0.042907715,0.08074951,0.00039434433,0.021835327,0.016555786,-0.038848877,-0.024047852,0.06274414,0.01499176,-0.0030841827,0.03338623,0.023727417,-0.07220459,0.022109985,0.0042419434,-0.019866943,0.01889038,0.012039185,0.025558472,-0.027160645,0.023452759,0.09869385,0.015670776,0.0018520355,-0.04486084,-0.0009112358,-0.023895264,0.0067443848,0.042114258,0.000975132,-0.009162903,-0.017944336,-0.015655518,-0.0047569275,0.02947998,-0.015701294,-0.033843994,-0.065979004,-0.017059326,0.03326416,-0.034362793,0.0049934387,0.0049095154,-0.012161255,-0.016601562,0.034362793,0.085998535,-0.012390137,0.045654297,0.01826477,-0.014526367,-0.014549255,0.12371826,-0.020065308,0.0037517548,-0.046173096,-0.053009033,0.015220642,0.07281494,-0.0046539307,0.026123047,-0.025741577,-0.017440796,0.024963379,-0.089904785,-0.023986816,-0.07366943,-0.0006928444,-0.026123047,-0.04675293,-0.009483337,-0.009681702,-0.0051994324,-0.0063438416,0.06652832,-0.04714966,-0.037384033,-0.095703125,0.029800415,0.011375427,-0.05130005,-0.018585205,0.0093307495,0.026641846,-0.02720642,-0.060180664,-0.01626587,0.029830933,-0.007411957,0.03857422,0.04473877,0.046142578,-0.012283325,-0.0413208,0.09472656,-0.046447754,0.013931274,0.051971436,-0.00021028519,-0.004508972,-0.07122803,-0.016738892,0.06585693,-0.004837036,-0.023239136,-0.033050537,0.026016235,0.0013494492,-0.0062828064,-0.03387451,-0.0033740997,0.0009613037,-0.027770996,-0.0036811829,0.022903442,-0.003545761,-0.027191162,-0.001247406,0.010398865,0.018676758,0.041778564,0.020736694,0.124694824,0.0084991455,-0.061187744,-0.008087158,-0.00091695786,-0.054534912,0.0031661987,-0.051849365,-0.04547119,0.046051025,-0.016555786,-0.027923584,-0.011787415,0.028121948,-0.010925293,0.052825928,-0.040161133,-0.01828003,0.044799805,0.034179688,-0.012718201,0.04159546,-0.0044937134,0.027801514,-0.03668213,0.03881836,0.05303955,-0.028289795,0.047973633,-0.019210815,0.013313293,0.013725281,0.029327393,-0.033111572,-0.008354187,0.016830444,0.0037651062,0.048980713,0.041778564,0.009498596,-0.086120605,-0.034210205,-0.06573486,-0.034057617,-0.021606445,0.01928711,-0.11206055,-0.020004272,-0.01360321,0.00970459,0.0035800934,0.06274414,-0.01524353,-0.024871826,-0.033081055,0.004749298,-0.029418945,-0.07647705,-0.02027893,-0.06149292,0.013832092,0.05014038,-0.0513916,-0.05206299,-0.04269409,0.06262207,0.028366089,0.06945801,0.02645874,0.00064086914,-0.05404663,-0.035583496,-0.02027893,0.0052719116,-0.049987793,0.07385254,-0.047027588,-0.043792725,0.030227661,0.012252808,-0.0034275055,0.036010742,0.033111572,-0.021270752,-0.021377563,0.09057617,-0.03225708,0.029296875,0.015220642,0.046447754,0.021148682,0.061676025,-0.1104126,0.0053596497,-0.03363037,0.02180481,-0.011695862,-0.056243896,-0.07775879,-0.046783447,-0.017807007,-0.010940552,0.001502037,0.055267334,0.00086688995,-0.006008148,-0.03036499,-0.01423645,-0.056030273,-0.011627197,-0.038238525,-0.018081665,0.037261963,-0.0018625259,-0.09649658,0.083496094,-0.04248047,-0.0031414032,-0.024658203,-0.04586792,-0.0059547424,-0.020492554,-0.042877197,-0.008102417,0.02722168,-0.016647339,0.008674622,-0.057128906,0.035247803,-0.017440796,0.051361084,0.043121338,-0.013397217,0.007686615,0.051818848,0.03161621,-0.029266357,-0.0010929108,-0.014404297,0.03918457,0.020462036,-0.03994751,0.01878357,-0.026504517,-0.015823364,-0.0104522705,-0.0030288696,-0.04714966,-0.0109939575,0.03652954,-0.0012578964,0.08001709,-0.015319824,-0.03363037,0.015960693,-0.022125244,-0.008781433,-0.016433716]'::vector) < '0.6'::double precision))
  Rows Removed by Filter: 168
Planning Time: 0.539 ms
Execution Time: 0.685 ms

If I'm reading this correctly, 10 rows are being returned and 168 are removed, and the distance query is being run on 10 + 168 = 178 rows i.e. all rows in the table. Is that correct?

rsomani95 commented 6 months ago

This might be the solution: https://www.postgresql.org/docs/current/queries-with.html

Will try this out and share updates.