Open tomplex opened 6 years ago
@tomplex I'm unsure what's up with pickling here. For what it's worth, I haven't used this in my own applications. Instead I've maintained a mapping of strings (like 'B0001'
) to ints and have used those ints as Rtree indexes (instead of the one generated by enumerate
). I recommend this as a workaround.
Can you provide a little more information?
I haven't seen any reports of problems with multiple indexes, but I can't rule it out.
Hey @sgillies, thanks for the response. I started out using rtree
the way you mention, but was just messing around with some other usage options as I'm trying to write a library around rtree
/ shapely
for my own use.
To answer your specific questions:
The app is using threading to get & load data into separate rtree
's simultaneously at the beginning, but there aren't ever multiple threads loading to a single index or reading from the same index. Here's a small example of what I mean:
from rtree import index
from threading import Thread
class Dataset:
def load(self, data):
self._rtree = index.Index(data)
def load_async(self, data):
self._thread = Thread(target=self.load, args=(data,))
self._thread.start()
d1 = Dataset()
d2 = Dataset()
d1.load_async(data1)
d2.load_async(data2)
# wait 'till it's done loading
results = d2.intersection('Polygon(...)')
I saw that the libspatialindex
library is not thread-safe for inserting or reading, but as far as I understood the comments on a few github issues this seemed like it was mostly an issue with multiple threads acting on single rtree
instances, and folks who were reporting issues were seeing things like segfaults and other catastrophic errors - not ones with pickle
. That said, I understand if your answer now is "don't use rtree
in any multithreaded context". =)
Hello,
I'm using
rtree
in an application I'm writing and running into some odd issues with the objects that I'm passing in to the index when creating it. Unfortunately I can't consistently replicate the issue (even with the same dataset), but I will provide an example of what I'm seeing in the hopes that someone has seen it before / can help me debug.I'm bulk-loading two indexes, each from a different generator of data, passing in the ID of each row as the
object
in the index, an alphanumeric primary key. It's in all cases a valid ascii string, with no fancy characters; just letters and numbers. The example below is contrived and not necessarily representative of the data I'm working with, which I unfortunately cannot share, but is a small scale example of what I'm doing.Now, when I perform intersections against the index, I've been using the
objects='raw'
argument, as I wish to get back from the index the IDs of the matching records:When doing this, I will inconsistently receive a number of similar but not identical errors:
I am not overriding the
loads
ordumps
methods of theRtree
(though I have tried that to resolve this issue). The inability to consistently get this error condition when running the same application with the same data has been frustrating. Could this issue be a result of my using multipleIndex
objects at once?Thanks in advance for any help.