Need to modify solr schema

[smadha]

As we are indexing millions of records I can see lot of issues with solr.

We initially did lot of handholding editing data types and using encoding but it's now constantly failing with OutOfMemoryError. I have tried increasing mem upto 1.5 gigs but it only gives us some extra time.

At this moment I am trying to create a new schema which addresses below limitations

• Instead of one single huge document we scale points across multiple documents
• We must be able to produce search results for a location. Query will be a LOCATION

I am planning below -

• one core in geoparser solr instance for every index
• one document in geoparser solr instance for each document in index
• one domain can cover multiple cores. core syntax will be -> domain_[id]

I am right now doing it only for indexed data on a new branch. As uploaded files thing works with no issues.

@MBoustani

[smadha]

New Branch - "new_solr_schema"

[smadha]

I have created a new admin core which will map input index to local core. Below is a example

ADMIN schema. This holds mapping information between input index and local solr cores
- id - domain-name
-   indexes -  list of indexes for this domain.
-   core_names - list of cores for those indexes.
-   point_list - list of num of points points found for each index
-   idx_size_list - list of size of target index for each index

{
        "core_names": [
          "test_1",
          "test_2"
        ],
        "point_len_list": [
          6,
          12883
        ],
        "id": "test",
        "idx_size_list": [
          388,
          7208416
        ],
        "indexes": [
          "http://host/solr/core1",
          "http://host/solr/core2"
        ]
}

• unique core in geoparser solr instance for every index. _In example test_1,test2
• one document in geoparser solr core for each document in index. _Core test1 will have as many docs as index http://host/solr/core1
• one domain can cover multiple cores. core syntax will be -> domain_[id]. _Domain test covers cores test_1,test2

@MBoustani @chrismattmann Do you guys agree?

[smadha]

Fixed in https://github.com/MBoustani/GeoParser/pull/49 Branch new_solr_schema merged to master