This issue is important because it fundamentally impacts how OWF shared TypeScript code is maintained and packaged for use in applications like the InfoMapper.

Because visualizations are components that may be used in multiple Angular applications, there needs to be a way to create modular code for visualizations so that the code in final applications is not written by copy and paste. Previous discussions have made progress on this but now we need to put something in place.

TSTool Background

TSTool is complex Java software that relies on layers of code in libraries, culminating with user interface that provides visualization capabilities. Some of the Java code has been ported to TypeScript to facilitate InfoMapper development. The following repository is the common code that contains many useful shared packages.

cdss-lib-common-java

In particular, under src/RTi are folders for the following. RTi is a hold-over from Riverside Technology Inc, which is a company that was paid to develop CDSS software. Steve used to work there.

DMI for data handling
GR for graphics (drawing)
GIS for spatial data handling and visualization
TS time series
GRTS visualization of time series
Util utility classes, under which are different packages

Relevant Java code has been ported to TypeScript to support InfoMapper development. The above folder structure has been retained in order to avoid confusion and to allow further coordination with Java code use.

Other Java libraries depend on the common library, as described in the README for the above. This layering of code has proven effective for 20 years of development history. Development in TSTool allows the source code for all libraries to be seen, so it is easy to change code anywhere. Angular demo application and InfoMapper are relying on npm to share, unless we implement a multiple repository approach similar to TSTool. Is that possible with Angular?

TypeScript code

Josh can provide more details about how TypeScript ported from Java is organized as well as other common code. It makes sense to clearly differentiate the Java packages so that TypeScript and Java code can continue to be coordinated. It probably also makes sense to have TypeScript-specific common code. The following needs to be determined:

Where in the source code for the applications does the shared Java-based code should live? Does the code that Josh ported from Java get migrated to demo application?
Is additional TypeScript-focused shared code developed and if so, how integrated is it with the Java-based code? Does it clearly have a separate package in the folder structure?
How easy will it be for Josh to enhance the above code when working on the InfoMapper? Or, will it be easier for Josh and Sofia to work on the demo application together and then periodically share code with InfoMapper? This seems more reasonable and would help put in place shared development protocols.

Organization of Visualization Code

Although it may be simpler to have one big npm package for all shared code and visualizations an hand off to InfoMapper, this results in a package that may not all be needed in applications. Therefore the following (or something like it) is recommended as organization for the visualization code. Each of the following packages correspond conceptually to a folder or zip file for npm, as discussed in the next section, and perhaps a top-level folder in the code tree.

owf-common-ng - package for common code (common code ported from Java and new TypeScript code)
owf-d3-common-ng - package for d3-related common code
owf-d3-gapminder-ng - package for a specific visualization, in this case Gapminder
owf-plotly-ng - package for plotly-related visualizations
owf-otherviz-ng - packages for each visualization package

The above will allow an application to pull in the common code and whichever visualization is needed. It is expected that the visualization code will need to be organized around third-party major visualization libraries. For example, in addition to general common code, there may need to be common code for interfacing with d3, plotly, leaflet, and other major packages. So, are additional packages needed for middle tiers or does that get bundled? This will probably depend on the complexity of a package and whether it makes sense to bundle visualizations. This determination will need to be made as development occurs.

npm packages

Previous discussions pointed out that Angular has libraries and components, etc. From a strictly npm perspective, the following provides information about packages:

https://docs.npmjs.com/about-packages-and-modules

If visualizations are developed in the demo application, they can be tested and released as one or more npm packages, for example zip files corresponding to each of the names listed above. These packages could be published for use by InfoMapper, for example using GitHub releases. If GitHub releases are not used, we could put in a shared location such as Amazon S3 storage similar to how other software installers are distributed.

Concerns

From experience with Java and TSTool, I have experience where doing development on an application often involves modifying library code and application in an iterative way. Having to package library code to hand off to an application can be painful and inefficient. For example, it might be painful if work is ongoing in the InfoMapper and have to wait on demo application. This inefficiency might be minimized if Josh and Sofia collaborate on developing visualization tools in the demo application and when ready, package and hand off to InfoMapper. It is important to decide on the primary development application and devote energy to that. For example, TSTool is an application where I develop most Java tools and then hand off to other applications via libraries.

Another concern is that demo application and InfoMapper may develop similar utility code that is different enough to not fit into a general common library. This needs to be evaluated as code is developed. Sometimes code can be made generic and sometimes it cannot. The code tends to evolve over time.

Because code might be changed in the demo application and then used in InfoMapper, there is potential that something may break. Therefore, need to think about automated testing. Programmers need to think about how to ensure that a downstream developer might be impacted by code changes. If something changes, how do we ensure that InfoMapper is not negatively impacted?

We cannot rely on manual testing and running InfoMapper to detect issues in shared code. Howe do we minimize issues?

As previously discussed, while packaging all shared code and visualizations to then hand off the the infomapper may be efficient, this may not be the case for all applications as many may not require the use of each visualization. This will result in unused code and will take up space in the development environment. Instead to resolve this each visualization package will live in a top level folder, potentially titled owf-libs, as a zip file.

Pending thoughts around the integration of these packages:

users may individually pick the packages they would want to use after cloning demo application
- they will integrate themselves following package's README
script could be created within build-util/ to copy all or a specific package to another application
- [argument could be provided to this script to specify package ]
- may also handle npm's linking utility and needed imports
Possible added feature to demo application that will the download of the package as a zip file
- EX: users navigate to demo application → navigate to gapminder demo → 'click here to download package' → package downloaded → user handles own integration using the package README

File Structure in consideration:

owf-app-viz-demo-ng/                      Git repository files for this repository.
├── build-util
│   ├── copy-gapminder-to-ng-app
│   ├── copy-owf-package-to-ng-app  [maybe this handles the linking and needed imports]
├── owf-libs/
│   ├── owf-lib-[packageName]-ng            zipped version of package ready for use
│   ├── owf-lib-[packageName]-[ng/js]
│   ├── owf-lib-[packageName]-[ng/js]
├── viz-demo-ng/                              The demo web application files
│   ├── src/
│   │   ├── app/
│   │   │   ├── OWF/                        Java Based OWF Packages
│   │   │   │   ├── DateTime 
│   │   │   │   ├── StringUtil
│   │   │   │   ├── TimeInterval
│   │   │   │   ├── TimeUtil
│   │   ├── assets/
│   │   │   ├── TSToolGraphDemo/
│   │   │   │   ├── [data-files]
│   │   │   │   ├── [configuration]

Addressing TypeScript Java Based Code:

From visualization examples such as TSTool Graphs in Plotly and ChartJS, some of the Java-based code has already been migrated into the demo application. These files were previously all located in a directory within assets/ and were used to facilitate visual demonstrations and to assist in prototyping features. Moving forward all of these Java-based packages will now live in an OWF folder within the src/app location along with the rest of the source code. This removes these Java ported code from assets, so that assets may contain only data, configuration, images and css files. Future visualization packages that require these Java ported dependencies will also contain this OWF/ folder.

EX: owf-lib-plotly-TSTool-ng.zip:

owf-lib-plotly-TSTool-ng
├── [library logic files]
├── node_modules/
├── OWF/
│   ├── DateTime 
│   ├── StringUtil
│   ├── TimeInterval
│   ├── TimeUtil
│   ├── ...
├── package.json
├── README.MD

Having the code ported from Java code live in a OWF folder will allow changes to be added to these packages by simply replacing this folder. NOTE: A version number may prove useful to keep track of modifications.

I would like a more complete mapping of OWF shared packages and discussion of process. More robust documentation is needed that can be included in the Demo Application repository, possibly being included in a MkDocs developer manual later.

Attached is my draft at this documentation. Please add to the repo and update with feedback as questions are answered. I will coordinate a call to discuss with the team. The issues and questions, once answered, should be converted to definitive statements in the documentation so it is clear how the issues are handled.

I will remind that Markdown files should include line breaks after punctuation so that line length is reasonably short (80-100 characters or less). Otherwise, using diff tools is not able to pin down differences as cleanly.

I had to attach a text file because Markdown files can't be attached.

angular-libraries.txt

These comments should be addressed by adding to the stand-alone Markdown documentation based on what I provided previously. Based on today's discussion, it seems that there is some question about what the top-level folder names are to support code for different packages and hierarchy of code within a package. Maybe it makes sense to track down some open source repositories to see what others are doing, for example Material and Angular showdown. It is difficult for me to recommend something when I am not an Angular developer and lack first-hand experience. The following are alternatives. In summary:

Do a better job of configuring tsconfig.json so that module resolution works no matter how the code is organized.
Set up the demo application as a "multi-project workspace" (see option 2 below).
Test npm packaging of code from the above with InfoMapper to make sure it works.
Will need to agree on folder and package names before finalizing the approach, once Angular way is understood.

Paths used to find classes in `src` and `node_modules`

Every programming language uses the concept of path to find code or library. In Java it is the "classpath". In Python it is the "python path". The C linker uses LD_LIBRARY_PATH. And, operating systems use PATH. Angular/typescript must be similar. For example, see the tsconfig.json Module Resolution properties such as paths.

Typescript uses import statements to import code. Ideally, such statements should rely on the paths to find the root of locations and then import statements should be relative to that. For example, if there is a library my-library, I don't necessarily care how the library is made available in development environment (in src/app or node_modules). I just want to access a class of interest. If the code is organized in a hierarchy, I should be forced to use the path to the specific class such as:

import { SomeClass } from 'ui/core/WindowManager.component';

This emphasizes that a specific class of interest is used (because the name SomeClass may be used multiple times in the code for different classes. And it also uses relative path that is transparent. See the following:

Simplifying TypeScript imports in Angular
Stack Overflow How to use paths in tsconfig.json
Material GitHub repository - how many npm packages does it get distributed? I see @angular/material in the package.json file so it appears that it is all distributed as one package?
Prime NG is also distributed as one npm package based on TriLynx applications.

So, the question is... how can code be organized so that the same import works for source code in one development environment and node_modules in another development environment, in particular when the code being used is organized in a hierarchical folder structure? It seems

Alternatives for organizing code hierarchy

One goal of organizing code into a hierarchy is to tame the anarchy of many files into some reasonable hierarchy that allows code to be shared and has logical groupings of files in different levels. The following discussion explores how to accomplish this goal. Regardless of approach, the code should be the same, with import statements that work regardless of source code solution and resulting node_modules.

Option 2 seems like something to investigate.

1. Single top-level folder

This is the default folder structure for an Angular application. See Angular workspace and project file structure.

src/app/owf/
   all the Java-ported code here, in folders matching Java
   ui/    -  window manager and dialog originally in InfoMapper?
      WindowManager.ts
      Dialog.ts
   viz/
      showdown/
         OWF showdown code here to process Markdown into HTML
         and optionally display in dialog that is managed in window manager
         Depends on the above general code.

The advantage of the above are:

All OWF code is under owf folder which helps with branding and isolating from other code that may exist.
Just need to add the one top-level folder to path to be found.
Simple if packaging all the code into one npm distribution.

Disadvantages are:

Mixing of code may make it more difficult to split out owf-common-ng and specific packages when packaged with npm.

2. Library-specific top-level folders

The following uses separate top-level folders. The folders under the main folder are intended to be used in import statements and therefore may seem redundant, but lead to clear code. After writing this initially, I found the "multiple projects" organization in the Angular workspace and project file structure documentation. The following is my rough cut but it should follow the Angular way of doing it.

src/app/owf-common/
   all the Java-ported code here, in folders matching Java
   ui/    -  window manager and dialog originally in InfoMapper?
      WindowManager.ts
      Dialog.ts
src/app/owf-showdown/
   viz/
      showdown/
         OWF showdown code here to process Markdown into HTML
         and optionally display in dialog that is managed in window manager
         Depends on the above general code.

The advantage of the above are:

Using separate folders makes it easier to align source code with separate npm packages.

Disadvantages are:

Need to have multiple folders in path for compiler to find classes.
Folder names are more complex.

3. Separate repositories

Taking the previous example one step further is that the code would live in separate repositories. This seems to be more work than is necessary at this point.

Viz-Angular/
   git-repos/
       owf-app-viz-demo-ng/             Main application repo
            viz-demo-ng/
               src/app/owf/                       Option 1
                   common/
               src/app/owf-common/       Option 2
       owf-common-ng/                     Separate common/shared code repo
           src/app/owf-common/
                   all the Java-ported code here, in folders matching Java
                   ui/    -  window manager and dialog originally in InfoMapper?
                      WindowManager.ts
                      Dialog.ts
       owf-showdown-ng/                     Separate component repo
           src/app/owf-showdown/
              viz/
                 showdown/
                     OWF showdown code here to process Markdown into HTML
                     and optionally display in dialog that is managed in window manager
                     Depends on the above general code.

The advantage of the above are:

Using separate repositories provides flexibility to develop code at different pace.
Clear alignment of repository to library to npm package.
Able to see code in applications rather than npm package hand-off via node_modules so able to work on code more efficiently.

Disadvantages are:

Separate repositories may require significant setup for even a small library. This may be a major disadvantage
Is it even possible for path in one Angular application to reference files in a different repository?

Considered File organization for the demo application.

add all demo applications to dialog-content folder similar to Infomapper structure
- each within the designated charting library folder
- Naming conventions: dialog-[chartingLibrary]-[vizName]
- 'dialog-chartjs-generic/'
- 'dialog-chartjs-snodas/'
- 'dialog-chartjs-TSgraph/'
add shared common code using owf folder (see below for more packaging details)
- will also hold future owf viz ng libraries

File Structure in consideration: [DEMO]

owf-app-viz-demo-ng/                                The application repository folder.
├── viz-demo-ng/                                    The application files.
│   ├── src/
│   │   ├── app/                                
│   │   │   ├── nav-bar                             General application components
│   │   │   ├── dialog-content/                     Top level folder for Dialog demos
│   │   │   │   ├── showdown/                       lib folder with demo components
│   │   │   |   │   ├── dialog-showdown/
│   │   │   │   ├── chartjs/                        lib folder with demo components
│   │   │   |   │   ├── dialog-chartjs-generic/
│   │   │   |   │   ├── dialog-chartjs-snodas/
│   │   │   |   │   ├── dialog-chartjs-snodas/
│   │   │   │   ├── plotly/                         lib folder with demo components
│   │   │   |   │   ├── dialog-plotly-generic/
│   │   │   |   │   ├── dialog-plotly-snodas/
│   │   │   |   │   ├── dialog-plotly-tsgraph/
│   │   │   |   │   ├── dialog-plotly-heatmap-generic/
│   │   │   |   │   ├── dialog-plotly-heatmap-csv-daily-streamflow/
│   │   │   │   ├── highcharts/                     lib folder with demo components
│   │   │   |   │   ├── dialog-highcharts-generic/
│   │   │   |   │   ├── dialog-highcharts-snodas/
│   │   │   |   │   ├── dialog-highcharts-tsgraph/
│   │   │   |   │   ├── dialog-highcharts-generic-heatmap/
│   │   │   |   │   ├── dialog-highcharts-hourly-heatmap/
│   │   │   │   ├── D3/                             lib folder with demo components
│   │   │   |   │   ├── dialog-d3-generic/
│   │   │   |   │   ├── dialog-d3-gapminder/
│   │   │   ├── owf/                                Shared code folder-dev happens here 
│   │   │   │   ├── classes/                        classes from java ported code
│   │   │   │   │   ├── DWR/
│   │   │   │   |   │   ├── stateMod/
│   │   │   │   |   │   ├── ...
│   │   │   │   │   ├── TS/
│   │   │   │   |   │   ├── DateValueTS
│   │   │   │   |   │   ├── ...
│   │   │   │   │   ├── ts-command-processor/
│   │   │   │   |   │   ├── ...
│   │   │   │   │   ├── Util
│   │   │   │   |   │   ├── ...
│   │   │   │   │   ├── ...
│   │   │   │   ├── viz/                             viz libraries                      
│   │   │   │   │   ├── owf-showdown
│   │   │   │   │   ├── owf-d3-gapminder
│   │   │   │   │   ├── owf-chartjs-snodas
│   │   │   │   │   ├── owf-highcharts-TSgraph-config
│   │   │   │   │   ├── owf-plotly-chartjs-TSgraph-config
│   │   ├── assets/
│   │   │   ├── css/
│   │   │   ├── showdownFiles/
│   │   │   │   ├── [markdown-files]
│   │   │   ├── TSGraphFiles/
│   │   │   │   ├── [data-files]
│   │   │   │   ├── [configuration]
│   │   │   ├── SnodasFiles/
│   │   │   │   ├── [data-files]
│   │   │   ├──...

Considered shared code folder organization:

Should be located undersrc/app:

currently located under src/app/map-components in the infomapper, would need to move out of map-components

This is the ideal file structure for this shared code:

── owf/                     
│   ├── common-classes/                         Java ported code 
│   │   ├── DWR/
│   |   │   ├── stateMod
│   |   │   ├── ...
│   │   ├── TS/
│   |   │   ├── DateValueTS
│   |   │   ├── ...
│   │   ├── Util
│   |   │   ├── ...
│   │   ├── ...
│   ├── viz/                        
│   │   ├── owf-showdown/
│   │   ├── owf-d3-gapminder/
│   │   ├── owf-chartjs-snodas/
│   │   ├── owf-plotly-TSgraph-config/      *references from certain common classes*
│   │   ├── owf-chartjs-TSgraph-config/     *references from certain common classes*

Naming Conventions: owf-[chartingLibrary]-[vizName]

This file structure implies having node modules package the top level owf folder and subfolders with its respected libraries.

From this stackoverflow question it may be possible to create an angular library with multiple modules . These exported modules can then be imported independently from the library.

owf would then be the ng library workspace with sub modules for each visualization package.

This solution will require some more research (answer is from 2017), but it may allow an owf ng lib workspace with multiple modules to export.

Briefly visited Createing Multiple Angular Libraries pt 1. Could provide a more up to date solution (Last update April 2020).

(Will revisit after first step of prototyping Showdown lib.)

Once owf package is ready to be shared with applications such as the infomapper, the following would be provided through node modules via symlink until ready to publicly distribute to npm.

owf-app-infomapper-ng/   The application repository folder.
  infomapper/            The application files.
    node_modules/        Library code, including packages imported from Demo Application.
      owf/               OWF packages from `npm` packages - is it a folder with sub-folders or multiple folders?
        *
      owf-common/
        DWR/
        TS/
        *
      viz/
        owf-chartjs-*/
        owf-plotly-*/
        *
    src/                 Source code, editable during development.
      app/               Application source code, under which all editable code exists (that is not npm package)
        *                Code specific to the application

See Angular Multiple project file structure. Can you (Sofia) evaluate how this would look for the demo app. For example, the Angular doc has:

my-workspace/
  ...             (workspace-wide config files)
  projects/       (generated applications and libraries)
    my-first-app/ --(an explicitly generated application)
      ...         --(application-specific config)
      e2e/        ----(corresponding e2e tests)
         src/     ----(e2e tests source)
         ...      ----(e2e-specific config)
      src/        --(source and support files for application)
    my-lib/       --(a generated library)
      ...         --(library-specific config)
      src/        --source and support files for library)

Would demo application look like:

owf-app-viz-demo-ng/
  ...             (workspace-wide config files)
  projects/       (generated applications and libraries)
    viz-demo-ng/ --(an explicitly generated application)
      ...         --(application-specific config)
      e2e/        ----(corresponding e2e tests)
         src/     ----(e2e tests source)
         ...      ----(e2e-specific config)
      src/        --(source and support files for application)
    owf-common/       --(a generated library)
      ...         --(library-specific config)
      src/        --source and support files for library)
         ts/
         util/
    owf-showdown/       --(a generated library)   DEPENDS ON owf-common
      ...         --(library-specific config)
      src/        --source and support files for library)

I also found the following. This helped confirm that doing the above is what we want to do. There may be better sources of information. Also, need to document what commands are run to implement what we are doing so we can learn from that in the future.

Specific comments on Sofia's folder structure, although not comprehensive are:

dialog-content is too specific. I recommend that if separate libraries are used for each visualization library that classes can exist in the main src folder of that library or subfolders like dialog. Is it possible to have a visualization display in a general form and then have a standard way of also including in a dialog? Then it would be flexible.
We should try to use lowercase for package paths, except classes can be MixedCase. Use d3 instead of D3. We could probably do that with Java ported classes such as util instead of Util.
For owf common code, I don't think classes folder is needed. Also, we can have both Java-ported and new general code in the same library. It should be simple enough to find the original Java based on the top-level folder.
I'll have more opinions on specific conventions when we start implementing code in libraries and folders because I can compare patterns to previous code that has been developed.
Once we figure this out for dialog-based visualization code, maybe Josh can do something similar in InfoMapper to turn the map components into a library. Then we can decide whether to also move that to the demo application. A first step would then be to make InfoMapper depend on libraries for general code and dialog-based visualizations but keep mapping components in InfoMapper. I don't want to totally break InfoMapper and it is going to be important to backup the repo and do a tag before embarking on anything major.
I'm not sure how namespace collisions are avoided. For example, the DateTime class was ported from Java to use with time series code. I can see that this class name would be used in other packages. If the tsconfig.json paths include more than one package that has DateTime in the same relative location to the node_modules/package/src folder, how would code be able to indicate that a specific version of DateTime should be used? In Java, the package path typically includes the organization. To provide this namespace in TypeScript, it would be necessary to use a src/owf folder in each library, but I don't see a discussion of that from what I found. Most of the time there is no top-level folder corresponding to an organization or namespace. Typescript does have namespaces but I did not dig into.

OpenWaterFoundation / owf-app-viz-demo-ng

Move InfoMapper code into demo app as the master copy for visualization components #27

TSTool Background

TypeScript code

Organization of Visualization Code

npm packages

Concerns

Paths used to find classes in `src` and `node_modules`

Alternatives for organizing code hierarchy

1. Single top-level folder

2. Library-specific top-level folders

3. Separate repositories

Considered File organization for the demo application.

Considered shared code folder organization:

OpenWaterFoundation / owf-app-viz-demo-ng

Move InfoMapper code into demo app as the master copy for visualization components #27

TSTool Background

TypeScript code

Organization of Visualization Code

npm packages

Concerns

Paths used to find classes in src and node_modules

Alternatives for organizing code hierarchy

1. Single top-level folder

2. Library-specific top-level folders

3. Separate repositories

Considered File organization for the demo application.

Considered shared code folder organization:

Paths used to find classes in `src` and `node_modules`