pinecone's filter solution

[ingberam]

@harsha-simhadri

running time (including index build): <5min

[harsha-simhadri]

on public queries:

pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '11.0', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,80919.5842261849,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.8774580000000001 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '13.0', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,73730.30551209228,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.8918290000000001 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '14.0', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,70934.77957185828,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.897807 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '14.3', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,69960.10194686487,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.89947 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '14.7', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,68993.96683759283,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.901645 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '15.0', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,68295.43433985539,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.9032640000000001 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '16.0', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,65807.0117827434,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.908244 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '17.0', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,63363.45523040706,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.912706 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '18.0', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,60615.84062975785,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.916847 pinecone,"pinecone_filter(('FilterIVFFlatU8', {'indexkey': 'FilterIVFFlatU8', 'num_clusters': '1024', 'precompute_intersection_threshold': '2500'}, {'fraction_coefficient': '19.0', 'fraction_exponent': '0.65', 'skip_clustering_threshold': 2000}))",yfcc-10M,10,58636.90129932696,0.0,139.2911684513092,9352640.0,0.0,0.0,filter,0.920562

[ingberam]

Thanks @harsha-simhadri for running. The results are as expected. Can you update the ongoing leaderboard? Thanks

[xiaofan-luan]

@ingberam Bigann should not be a competition for benchmarketing. The main purpose of this competition is for everyone to share their understanding of vector retrieval through open-source code, and to exchange ideas. Pinecone should open source their code, otherwise, it is cheating from my perspective.

[harsha-simhadri]

@ingberam Bigann should not be a competition for benchmarketing. The main purpose of this competition is for everyone to share their understanding of vector retrieval through open-source code, and to exchange ideas. Pinecone should open source their code, otherwise, it is cheating from my perspective.

@xiaofan-luan Since you bring this up and use the "cheating" word:

1. Let us decide whether companies can participate or not. We had entries from Baidu and Zilliz affiliated authors in NeurIPS'23. The Zilliz affiliated author (Zihao Wang) submitted to NeurIPS'23 without declaring the correct affiliation. When questioned, there was no response for a long time. We let this slide. Zilliz was represented on the organizing committee and knew the rules.
2. On OSS vs closed source, my vote is for OSS. The organizers can take a vote and enforce in official competitions. You or your team's representative is welcome to be involved in future effort and enforce rules uniformly. Outside of competitions, why stop people from using the framework the way they want?

[harsha-simhadri]

Thanks @harsha-simhadri for running. The results are as expected. Can you update the ongoing leaderboard? Thanks

@ingberam
Happy to merge. As we discussed, keeping an ongoing leaderboard is a lot of effort without automation and more involvement in running code from others. I do not have time to do this in the next 6 months. Please consider contributing automation if you have time. We should also clearly mark OSS and closed source solutions on such a leaderboard. Thanks!

[ingberam]

Thanks @harsha-simhadri for running. The results are as expected. Can you update the ongoing leaderboard? Thanks

@ingberam Happy to merge. As we discussed, keeping an ongoing leaderboard is a lot of effort without automation and more involvement in running code from others. I do not have time to do this in the next 6 months. Please consider contributing automation if you have time. We should also clearly mark OSS and closed source solutions on such a leaderboard. Thanks!

Sure, I can help with evaluating ongoing submissions. I'll open an issue to track the automatic eval topic. +1 on marking OSS/vs closed source, can be done as a footnote on the plot stating that pinecone (and HWTL_SDU-ANNS) is not open source.

[xiaofan-luan]

@ingberam Bigann should not be a competition for benchmarketing. The main purpose of this competition is for everyone to share their understanding of vector retrieval through open-source code, and to exchange ideas. Pinecone should open source their code, otherwise, it is cheating from my perspective.

@xiaofan-luan Since you bring this up and use the "cheating" word:

1. Let us decide whether companies can participate or not. We had entries from Baidu and Zilliz affiliated authors in NeurIPS'23. The Zilliz affiliated author (Zihao Wang) submitted to NeurIPS'23 without declaring the correct affiliation. When questioned, there was no response for a long time. We let this slide. Zilliz was represented on the organizing committee and knew the rules.
2. On OSS vs closed source, my vote is for OSS. The organizers can take a vote and enforce in official competitions. You or your team's representative is welcome to be involved in future effort and enforce rules uniformly. Outside of competitions, why stop people from using the framework the way they want?

Thanks for the response.

My use of 'cheating' might have been too strong a term, but it relates to the fact that, during the competition, there was no disclosure of this leaderboard, nor was it explicitly stated that NeurIPS would accept submissions of closed-source code for evaluation. While an employee from Zilliz did participate in the competition, it is just a side work during leisure time and we did not officially attribute this participation as a credit to Zilliz through any formal channels. If this has led to any misunderstandings, we are prepared to retract our submission related to pyanns.

Additionally, we possess a faster internal solution than pyanns submitted, which has not been open sourced due to business reasons. On a personal note, I propose that we launch a new benchmark rather than list as a continuation of the competition. We would be eager to submit our closed-source results in this leaderboard. But for BigANN competitions, I personally believe the most crucial aspect is to inspire others. Merely providing a score without sharing the code and detailed explanations seems to contribute little to positive changes in the industry.

[harsha-simhadri]

@ingberam Bigann should not be a competition for benchmarketing. The main purpose of this competition is for everyone to share their understanding of vector retrieval through open-source code, and to exchange ideas. Pinecone should open source their code, otherwise, it is cheating from my perspective.

@xiaofan-luan Since you bring this up and use the "cheating" word:

1. Let us decide whether companies can participate or not. We had entries from Baidu and Zilliz affiliated authors in NeurIPS'23. The Zilliz affiliated author (Zihao Wang) submitted to NeurIPS'23 without declaring the correct affiliation. When questioned, there was no response for a long time. We let this slide. Zilliz was represented on the organizing committee and knew the rules.
2. On OSS vs closed source, my vote is for OSS. The organizers can take a vote and enforce in official competitions. You or your team's representative is welcome to be involved in future effort and enforce rules uniformly. Outside of competitions, why stop people from using the framework the way they want?

Thanks for the response.

My use of 'cheating' might have been too strong a term, but it relates to the fact that, during the competition, there was no disclosure of this leaderboard, nor was it explicitly stated that NeurIPS would accept submissions of closed-source code for evaluation. While an employee from Zilliz did participate in the competition, it is just a side work during leisure time and we did not officially attribute this participation as a credit to Zilliz through any formal channels. If this has led to any misunderstandings, we are prepared to retract our submission related to pyanns.

If as you claim, Zihao Wang submitted on his personal time, what authority do you have over him to make him withdraw his "personal" submission?

Additionally, we possess a faster internal solution than pyanns submitted, which has not been open sourced due to business reasons. On a personal note, I propose that we launch a new benchmark rather than list as a continuation of the competition. We would be eager to submit our closed-source results in this leaderboard. But for BigANN competitions, I personally believe the most crucial aspect is to inspire others. Merely providing a score without sharing the code and detailed explanations seems to contribute little to positive changes in the industry.

The NeurIPS'23 competition and leaderboard is closed. I created a tag to indicate the state of the repo as of the competition conclusion.

We are talking about a new ongoing leaderboard. Amir seems interested in contributing to this. You are welcome to collaborate with Amir and make a proposal for 2024. We can separate OSS and closed source solutions and leaderboards.

On this forum, as you say, it is more constructive to create a framework for research. So please consider adding value by creating new datasets or adding automation or commentary. Thanks!

[harsha-simhadri]

Thanks @harsha-simhadri for running. The results are as expected. Can you update the ongoing leaderboard? Thanks

@ingberam Happy to merge. As we discussed, keeping an ongoing leaderboard is a lot of effort without automation and more involvement in running code from others. I do not have time to do this in the next 6 months. Please consider contributing automation if you have time. We should also clearly mark OSS and closed source solutions on such a leaderboard. Thanks!

Sure, I can help with evaluating ongoing submissions. I'll open an issue to track the automatic eval topic. +1 on marking OSS/vs closed source, can be done as a footnote on the plot stating that pinecone (and HWTL_SDU-ANNS) is not open source.

Thanks, I will merge your PRs and will look forward to automation on leaderboard update for PRs!

[xiaofan-luan]

@ingberam Bigann should not be a competition for benchmarketing. The main purpose of this competition is for everyone to share their understanding of vector retrieval through open-source code, and to exchange ideas. Pinecone should open source their code, otherwise, it is cheating from my perspective.

@xiaofan-luan Since you bring this up and use the "cheating" word:

1. Let us decide whether companies can participate or not. We had entries from Baidu and Zilliz affiliated authors in NeurIPS'23. The Zilliz affiliated author (Zihao Wang) submitted to NeurIPS'23 without declaring the correct affiliation. When questioned, there was no response for a long time. We let this slide. Zilliz was represented on the organizing committee and knew the rules.
2. On OSS vs closed source, my vote is for OSS. The organizers can take a vote and enforce in official competitions. You or your team's representative is welcome to be involved in future effort and enforce rules uniformly. Outside of competitions, why stop people from using the framework the way they want?

Thanks for the response. My use of 'cheating' might have been too strong a term, but it relates to the fact that, during the competition, there was no disclosure of this leaderboard, nor was it explicitly stated that NeurIPS would accept submissions of closed-source code for evaluation. While an employee from Zilliz did participate in the competition, it is just a side work during leisure time and we did not officially attribute this participation as a credit to Zilliz through any formal channels. If this has led to any misunderstandings, we are prepared to retract our submission related to pyanns.

If as you claim, Zihao Wang submitted on his personal time, what authority do you have over him to make him withdraw his "personal" submission?

Additionally, we possess a faster internal solution than pyanns submitted, which has not been open sourced due to business reasons. On a personal note, I propose that we launch a new benchmark rather than list as a continuation of the competition. We would be eager to submit our closed-source results in this leaderboard. But for BigANN competitions, I personally believe the most crucial aspect is to inspire others. Merely providing a score without sharing the code and detailed explanations seems to contribute little to positive changes in the industry.

The NeurIPS'23 competition and leaderboard is closed. I created a tag to indicate the state of the repo as of the competition conclusion.

We are talking about a new ongoing leaderboard. Amir seems interested in contributing to this. You are welcome to collaborate with Amir and make a proposal for 2024. We can separate OSS and closed source solutions and leaderboards.

On this forum, as you say, it is more constructive to create a framework for research. So please consider adding value by creating new datasets or adding automation or commentary. Thanks!

Fair enough, we'd like to create github actions to automate the check in process if necessary. Even if close source solution is listed in a separate leader board, I guess it may still very reasonable to have short description about each algorithm otherwise this will soon become a game of numbers

[harsha-simhadri]

@ingberam Bigann should not be a competition for benchmarketing. The main purpose of this competition is for everyone to share their understanding of vector retrieval through open-source code, and to exchange ideas. Pinecone should open source their code, otherwise, it is cheating from my perspective.

@xiaofan-luan Since you bring this up and use the "cheating" word:

1. Let us decide whether companies can participate or not. We had entries from Baidu and Zilliz affiliated authors in NeurIPS'23. The Zilliz affiliated author (Zihao Wang) submitted to NeurIPS'23 without declaring the correct affiliation. When questioned, there was no response for a long time. We let this slide. Zilliz was represented on the organizing committee and knew the rules.
2. On OSS vs closed source, my vote is for OSS. The organizers can take a vote and enforce in official competitions. You or your team's representative is welcome to be involved in future effort and enforce rules uniformly. Outside of competitions, why stop people from using the framework the way they want?

Thanks for the response. My use of 'cheating' might have been too strong a term, but it relates to the fact that, during the competition, there was no disclosure of this leaderboard, nor was it explicitly stated that NeurIPS would accept submissions of closed-source code for evaluation. While an employee from Zilliz did participate in the competition, it is just a side work during leisure time and we did not officially attribute this participation as a credit to Zilliz through any formal channels. If this has led to any misunderstandings, we are prepared to retract our submission related to pyanns.

If as you claim, Zihao Wang submitted on his personal time, what authority do you have over him to make him withdraw his "personal" submission?

Additionally, we possess a faster internal solution than pyanns submitted, which has not been open sourced due to business reasons. On a personal note, I propose that we launch a new benchmark rather than list as a continuation of the competition. We would be eager to submit our closed-source results in this leaderboard. But for BigANN competitions, I personally believe the most crucial aspect is to inspire others. Merely providing a score without sharing the code and detailed explanations seems to contribute little to positive changes in the industry.

The NeurIPS'23 competition and leaderboard is closed. I created a tag to indicate the state of the repo as of the competition conclusion. We are talking about a new ongoing leaderboard. Amir seems interested in contributing to this. You are welcome to collaborate with Amir and make a proposal for 2024. We can separate OSS and closed source solutions and leaderboards. On this forum, as you say, it is more constructive to create a framework for research. So please consider adding value by creating new datasets or adding automation or commentary. Thanks!

Fair enough, we'd like to create github actions to automate the check in process if necessary. Even if close source solution is listed in a separate leader board, I guess it may still very reasonable to have short description about each algorithm otherwise this will soon become a game of numbers

That is a reasonable proposal. Can you please agree on such rules with Amir and others on this forum and write down the new rules and evaluation cadence on a separate folder "2024". @ingberam suggested quarterly sweeps of PRs and might volunteer to drive the first quarter.