Research specifications of Amazon CloudSearch and define initial milestones of croonga project

[piroor]

Current state

No milestone.

Expected state

There are clear milestones. They are small enough to release the initial version for people rapidly.

How to solve

1. Research the documentation of Amazon CloudSearch, to make them clear; how to upload data and select records.
2. Define minimal milestones.

[piroor]

ACS accepts "Search Data Format (SDF)" as the input. http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/SvcConcepts.SDF.html

SDF data can be sent as JSON or XML.

• http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/DocumentsBatch.JSON.html#DocumentsBatch.JSON.Requests
• http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/DocumentsBatch.XML.html#DocumentsBatch.XML.Requests

People send the data to the web API via POST method (or the console).

http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/senddata.html

This can be mapped to Groonga's load command.

[piroor]

Before uploading data, we have to configure indexes of fields. http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/configureindexfields.API.html This can be mapped to Groonga's create_index command.

[piroor]

Before creating indexes, we have to create domains. http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/creatingdomains.html#createdomain.API This can be mapped to Grooga's create_table command.

[piroor]

To search in stored data, we have to call Web API by GET method. http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/searching.html This can be mapped to Groonga's select command.

[piroor]

Draft of the initial milestone:

• People can upload JSON SDF via Web API (POST method). This is mapped to the load command.
• People can search documents via Web API (GET method). This is mapped to the select command.

Dropped topics from the initial milestone:

• We don't support CreateDomain and ConfigureIndexFields APIs yet. People have to create and configure tables via groonga commands, not croonga.
• We don't support XML SDF yet.

[piroor]

The search API has following features:

• Embed fragments of indexed documents to the response http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/retrievingdata.html
• Results in XML http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/gettingxmlresults.html
• Sort by text_relevance scores, alphabetical, numerical, or user defined order. http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/sortingresults.html
• Pagination ("limit" and "offset") http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/pagination.html

[piroor]

Draft v2 of the initial milestone:

• People can upload JSON SDF via Web API (POST method). This is mapped to the load command.
• People can search documents via Web API (GET method). This is mapped to the select command.
  • People can get the search results as a JSON.
  • Only top 10 fields are returned.
  • croonga will return all fields always.

Dropped topics from the initial milestone:

• We don't support CreateDomain and ConfigureIndexFields APIs yet. People have to create and configure tables via groonga commands, not croonga.
• We don't support XML SDF yet.
• We don't support customizing fields, pagination, sorting, and XML-based response of the search API yet.

---

Draft of the 2nd milestone:

• Add supports about customizable fields, pagination, and sorting for the search API.

[piroor]

SDF can have its type from two of types: add and delete. http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/DocumentsBatch.JSON.html#DocumentsBatch.JSON.Requests

[piroor]

Draft v3 of the initial release:

• People can upload SDF via Web API (POST method). This is mapped to the load command.
  • Only JSON format is acceptable.
  • Only "add" method is available in the batch.
• People can search documents via Web API (GET method). This is mapped to the select command.
  • People can get the search results as a JSON.

Dropped topics from the initial release:

• We don't support CreateDomain and ConfigureIndexFields APIs yet. People have to create and configure tables via groonga commands, not croonga.
• We don't support XML SDF yet.
• We don't support "delete" type batch yet.
• We don't support XML responses yet.

---

milestone 1:

• Implement the batch API with minimal features.

milestone 2:

• Implement the search API with minimal features.
  • Only top 10 fields are returned.
  • croonga will return all fields always.
  • We don't support customizing fields, pagination, sorting, and XML-based response of the search API yet.

milestone 3:

• Add supports about customizable fields, pagination, and sorting for the search API.

[piroor]

We should support CreateDomain and ConfigureIndexFields APIs on the initial release.

[piroor]

Draft v4 of the initial release:

• People can upload SDF via Web API (POST method) "batch". http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/DocumentsBatch.html This is mapped to the load command.
  • Only JSON format is acceptable.
  • Only "add" method is available in the batch.
• People can search documents via Web API (GET method) "search". http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/Search.html This is mapped to the select command.
  • People can get the search results as a JSON.
• People can create domain via Web API (GET method) "Configuration API, Action=CreateDomain". http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/API_CreateDomain.html This is mapped to "-n" option (creating new database), and table_create commands.
  • Request Authentication is not required.
• People can configure indexes via Web API (GET method) "Configuration API, Action=DefineIndexField". http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/API_DefineIndexField.html This is mapped to the column_create command.
  • Request Authentication is not required.

Dropped topics from the initial release:

• We don't support authentication of some dangerous requests (CreateDomain and DefineIndexField) yet.
• We don't support XML SDF yet.
• We don't support "delete" type batch yet.
• We don't support XML responses yet.

---

milestone 1:

• Define templates for schemers of the database used by croonga.

milestone 2:

• Implement the batch API with minimal features.
  • We don't support CreateDomain and DefineIndexField actions yet. People have to create and configure tables via groonga commands, not croonga.

milestone 3:

• Implement the search API with minimal features.
  • Only top 10 fields are returned.
  • croonga will return all fields always.
  • We don't support customizing fields, pagination, sorting, and XML-based response of the search API yet.

milestone 4:

• Add supports about customizable fields, pagination, and sorting for the search API.

milestone 5:

• Implement Configuration API, Action=DefineIndexField.

milestone 6:

• Implement Configuration API, Action=CreateDomain. We have to create the database via groonga command.

milestone 7:

• Create the database automatically via Configuration API, Action=CreateDomain.

[piroor]

milestone 7:

• Create the database automatically via Configuration API, Action=CreateDomain.

This has been already done.

[piroor]

http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/creatingdomains.html

Restrictions about search domains (tables):

• Domain name must be unique.
• The first letter must be [a-z0-9]. Rest letters can be [-a-z0-9].
  • Upper case letters, underscore, and other special characters are not allowed.
• A domain name must be 3 or more than 28 characters long.

[piroor]

http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/configureindexfields.html

Restrictions about column names (index fields):

• Allowed characters for the name: [a-z0-9_]
  • Upper case letters, hyphen, and other special characters are not allowed.
• A column name must be 3 or more than 64 characters long.
• Reserved column names cannot be used. Known reserved names are:
  • body
  • docid
  • text_relevance

[piroor]

For the initial release, I think we should define a model case. My plan:

• A basic address book named "companies"
• There are some columns: "name", "address", "email_address" (including one underscore), "description", and so on.
• There are some test data based on public information: "ClearCode Inc.", and so on.

Based on such a model case, we can define a schemer for Groonga and develop two APIs batch and search parallely.

[piroor]

http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/configreq.html

ACS accepts both GET and POST methods for the configuration API. On the other hand, search API accepts only GET, and batch API accepts only POST.

[piroor]

http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/creatingsdf.html

Restrictions of batches:

• For the batch API, we must give unique ID for each indexed document (=docid).
  • Allowed pattern for a docid: [a-z0-9][a-z0-9_]*
• Each document must have its version. We must specify the new version of the document when adding/updating new data.
  • For example: add(version=1) => add(version=2):update => add(version=3):update, delete(version=4)
• "null" is invalid value for any column.
• Each batch request must be smaller than 5MB. And, one document must be smaller than 1MB.

[piroor]

http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/addsourcefield.html

When you create a new index, you can specify the "source" of the index. "Copy" is just same to groonga's default behavior. Others ("Trim Title" and "Map") are like "stored procedure"s. On the initial release, I think we should drop support of "Trim Title" and "Map".

[piroor]

http://docs.amazonwebservices.com/cloudsearch/latest/developerguide/indexing.html

ACS can rebuild search indexes with new configurations, with documents already updated. In other words, ACS stores the original document separately from the indexed database. This is very different from groonga.

[piroor]

Saved as a wiki page. Roadmap => Initial Release