[RFC] Flint main class refactor

[noCharger]

Is your feature request related to a problem?

The implementation of FlintREPL and FlintJob classes within Flint tightly couples the storage layer (OpenSearch) with the core functionalities that manage Streaming and Interactive Jobs, such as the queue service and heartbeat publish service. Given OpenSearch's flexibility to serve various roles, this tight coupling does not effectively segregate job control logic from query execution logic. As a result, it complicates error message exposure and thread management, potentially leading to less maintainable code and increased difficulty in troubleshooting.

What solution would you like?

The proposed refactoring involves creating new interfaces to decouple the session and command management functionalities from the OSClient. Here are the key points for Interactive Jobs:

• Flint Session Model and Flint Command Model should be extensible with context map
• REPLSessionManager and CommandLifecycleManager Interfaces should be provided
  • SessionManager trait: Abstracts the session management functionalities, providing methods for retrieving session details, updating session details, recording heartbeats, and checking if session is free to process query
  • CommandLifecycleManager trait: Abstracts the command lifecycle management functionalities, providing methods for preparing the command lifecycle, initializing and finalizing the command lifecycle, checking for pending commands, and updating command details.
• Introduce an interface for query result writer, which takes dataframe and other metadata
• Refactoring Changes:
  • Removal of flintSessionIndexUpdater: The session index updating functionalities will be integrated directly into the methods where they are needed, streamlining the update process.
  • Adoption of FlintReader for Read Operations: The existing OpenSearchQueryReader will be replaced by FlintReader under the new interface structure, making it more generic and flexible for handling different data sources or backend systems.

[TODO: @seankao-az ] will add flint indicies related key points:

[vamsi-amazon]

can you add some background of existing classes whenever you get time. @noCharger