go-ego / riot

Go Open Source, Distributed, Simple and efficient Search Engine; Warning: This is V1 and beta version, because of big memory consume, and the V2 will be rewrite all code.
Apache License 2.0
6.11k stars 473 forks source link

[Question or PR] Can tow engine share the same segmenter at the same time? #51

Closed szyhf closed 6 years ago

szyhf commented 6 years ago

Description

For some reason I have to use more than one engine at the same process, but while init each engine have to load the dictionary for segment, that cost a lot of time and memory, if I share the segmenter for different engine, will that case the concurrent problem?

If it can be done, may I make a PR for this engine option? ...

szyhf commented 6 years ago

Looks like the opt.NotUseGse can't work properly while use storage.

load storage will call innerIndexDoc and send doc to segment chan.

vcaesar commented 6 years ago

I will test update it, and welcome to optimization.

szyhf commented 6 years ago

All right, I'll try

szyhf commented 6 years ago

@vcaesar By the way, really pleasure to see the update of dep

szyhf commented 6 years ago

@vcaesar quite busy in past days, and our project fix the share with a temp solution, since the segmenter has a complex struct/interface and by your design I felt that you want keep the config simple.

I'm wondering using some design like what gRPC using in callOption`dialOption` to inject the complex implement like below:

// NewServer creates a gRPC server which has no service registered and has not
// started to accept requests yet.
func NewServer(opt ...ServerOption) *Server {
    //
}

What I Image the implement may look like:

type Optioner interface{
  // not designed yet
}

func (engine *Engine) Init(options types.EngineOpts, otherOpts ... Optioner ){

}
opt := types.EngineOpts{
//...
}
seg := mySeg{} // implement a `Optioner` interface
sthComplexToo := sth{} // implement a `Optioner` interface, too, may be the self imp storager
this.riotEngine = new(riot.Engine)
riotEngine.Init(opt, seg, sthComplexToo)

What's your opinion? My first target is to stripping the segmenter and second is the storager.

vcaesar commented 6 years ago

Thanks for your tips and contributions, but I have a better solution, please see the code #55 .

szyhf commented 6 years ago

@vcaesar Great ~