This is a major refactor of the lectern client code. The starting point was the code as copied from js-lectern-client. The objective of the refactor is to:
Use Lectern Dictionary types throughout the Lectern Client
Remove type assertions and use of any types
Organize code into logical modules
Remove unused or out of scope modules (parallelization solution)
Keep the package exports as unchanged as possible (may be changed in future refactoring steps)
Code Reorganization
Actions Taken on Original Code
src/change-analyzer.ts: Provides a different structure for reviewing changes retrieved from the lectern server diff between schemas
Code is kept, moved into directory src/changeAnalyzer
Appears to be built for ARGO project, specific ARGO dictionary meta properties were removed from this work
This whole module should be considered for removal in the future
src/schema-entities.ts: Types to support client, most were duplicate of the shared lectern dictionary types
Removed duplicate lectern dictionary types
Unique client types were about the data submitted for validation, these were moved to src/types/dataRecord.ts
Schema validation error types were relocated alongside the validation code
Change Analysis types were moved alongside the changeAnalyzer code
src/schema-functions.ts: This included code for processing raw TSV data, organizing validation pipelines, and all the validation logic for each restriction type
the main exports here were the raw data processing functions, these were moved to a module src/processing
the remaining code is mostly related to validating data and utility functions to assist in this, this was reorganized into several sub-modules in src/validation
src/schema-error-messages.ts: Code used for generating the error content for validation and processing failures
Error message formats were strictly typed for each validation type in src/validation/types/validationErrorTypes.ts
Functions for generating each type of error message now live alongside the validation code
src/schema-rest-client.ts: Data fetching from a Lectern server
Remained mostly unchanged, move to src/rest/index.ts
src/utils.ts: Type checking functions and a few other formatting utilities
Left mostly unchanged in the same location
Checks namespace and functions deleted, all references to this were removed
src/parallel.ts and src/schema-worker.ts: Node server solution for running the process function in a worker thread
These were completely removed as out of scope for the client. Server's are free to write their own functionality for running processing in parallel or in the main thread, as they need. The client has the objective of being useful in any server (not just NodeJS) and also in the browser.
src/changeAnalysis: Processing for schema diffs retrieved from Lectern Server. Appears to be added for ICGC ARGO. Consider for removal from Lectern Client unless a general need for this is identified.
src/processing: Raw data processing. Exports the functions that raw TSV data is passed to to be checked vs a Lectern dictionary. Relies heavily on the validation module.
src/rest: Rest client to fetch dictionaries and diffs from Lectern. This can be heavily expanded in future work to interact with all of Lectern's REST features.
src/types: TS types to represent the TSV data records provided for validation.
src/validation: Functionality to validate data records.
src/validation/dataRecordValidation: Validation code for unprocessed data records. Includes checks such as confirming the all columns are provided, no unrecognized columns are provided, and that the data provided is the expected type (string/number/boolean, an array, etc.)
src/validation/fieldRestrictions: Validation code for a single field in a data record. Includes checks such as Regex, Range, Scripts etc.
src/validation/schemaRestrictions: Validation code for an entire schema or data dictionary. Includes the UniqueKey and ForeignKey checks
src/validation/types: Validation function structure and error response types
src/validation/utils: Utilities for interacting with data records used by several validation functions
This is a major refactor of the lectern client code. The starting point was the code as copied from js-lectern-client. The objective of the refactor is to:
any
typesCode Reorganization
Actions Taken on Original Code
src/change-analyzer.ts
: Provides a different structure for reviewing changes retrieved from the lectern server diff between schemassrc/changeAnalyzer
src/schema-entities.ts
: Types to support client, most were duplicate of the shared lectern dictionary typessrc/types/dataRecord.ts
src/schema-functions.ts
: This included code for processing raw TSV data, organizing validation pipelines, and all the validation logic for each restriction typesrc/processing
src/validation
src/schema-error-messages.ts
: Code used for generating the error content for validation and processing failuressrc/validation/types/validationErrorTypes.ts
src/schema-rest-client.ts
: Data fetching from a Lectern serversrc/rest/index.ts
src/utils.ts
: Type checking functions and a few other formatting utilitiesChecks
namespace and functions deleted, all references to this were removedsrc/parallel.ts
andsrc/schema-worker.ts
: Node server solution for running theprocess
function in a worker threadNew Code Structure
src/changeAnalysis
: Processing for schema diffs retrieved from Lectern Server. Appears to be added for ICGC ARGO. Consider for removal from Lectern Client unless a general need for this is identified.src/processing
: Raw data processing. Exports the functions that raw TSV data is passed to to be checked vs a Lectern dictionary. Relies heavily on thevalidation
module.src/rest
: Rest client to fetch dictionaries and diffs from Lectern. This can be heavily expanded in future work to interact with all of Lectern's REST features.src/types
: TS types to represent the TSV data records provided for validation.src/validation
: Functionality to validate data records.src/validation/dataRecordValidation
: Validation code for unprocessed data records. Includes checks such as confirming the all columns are provided, no unrecognized columns are provided, and that the data provided is the expected type (string/number/boolean, an array, etc.)src/validation/fieldRestrictions
: Validation code for a single field in a data record. Includes checks such as Regex, Range, Scripts etc.src/validation/schemaRestrictions
: Validation code for an entire schema or data dictionary. Includes the UniqueKey and ForeignKey checkssrc/validation/types
: Validation function structure and error response typessrc/validation/utils
: Utilities for interacting with data records used by several validation functions