search

lanl / 4DGB

4D Genome Browser project
1 stars 2 forks source link

Proposal: Re-organizing directory layout #14

Closed camtauxe closed 3 years ago

camtauxe commented 3 years ago

Since I was first brought on to this project, I've been concerned that the way our directories are organized is hard to follow. This project contains three distinct pieces of software: A browser-based Javascript library, a python library, and a python web application. All three of these are closely-coupled together, so I still think it's a good idea to keep them all in the same repository like this, but their code right now is all inter-mingling, which makes it hard to understand what's going on.

Currently, our directory layout looks this:

bin (build scripts )
doc ( API documentation )
img ( images used in documentation )
projects ( project source data )
release ( gitignored, release-able builds )
schema ( project.json schema)
src
    - gentk ( python library code )
    - gtk
        - css/img/js ( static web content )
        - py ( flask server code )
    - img ( more images? )
    - settings ( configuration templates for Flask )
    - third_party ( copy of Three.JS )
    - ( more static web content )
- testing (test code)
- ( the usual readme, package.json, requirements.txt stuff )

I propose we rearrange our project to look like this:

bin (build scripts )
doc ( API documentation )
    - img ( images used in documentation )
    - schema ( project.json schema)
projects ( project source data )
release ( gitignored, release-able builds )
server
    - static ( static web content, except for JS library )
    - ( flask server code )
client-py
    - ( python library code )
client-js
    - ( JS library code )
- testing (test code)
- ( the usual readme, package.json, requirements.txt stuff )

You may notice that there isn't a space for the Three.JS code anymore. That's because as a part of this proposal, I think we should refactor the Javascript library into a module that can be built with webpack, allowing us to distribute the library as a single, minified Javascript file that has Three.JS, and any other dependencies automatically rolled in. This would also allow us to write tests for the library. The build step for webpack can easily be added to the make_release script, so the build process doesn't get any more complicated.

My hope is that, by more cleanly separating the different components of the project, and by more clearly labeling directories, it'll be easier for newcomers to understand what's going on, as well as easier for all of us to work on separate components without stepping on each others' toes.

I'd like to hear the others' feedback on this. If you like, I can create a separate branch in my fork with these changes so we can try it out.

dhrogers commented 3 years ago

HI, Cameron -

Thanks for taking the time to think about the re-org. This looks good to me.

One minor thing is that we should include the schema.json file somewhere in the source, as it's used now by one of the scripts to validate a project.json file. When we do a release, this file should be posted as part of the release, so it can be used to validate schemas. Not sure where this should go.

Also, we could flatten out the src/gtk directories by moving them up into the source, if that makes sense.

I'd say that we do a code freeze at the end of the day today, and let you do a branch for the re-org. I'll be working on the presentation most of Wed, so I won't be coding. Ricardo can check in his changes by COB today.

Thoughts?

Since I was first brought on to this project, I've been concerned that the way our directories are organized is hard to follow. This project contains three distinct pieces of software: A browser-based Javascript library, a python library, and a python web application. All three of these are closely-coupled together, so I still think it's a good idea to keep them all in the same repository like this, but their code right now is all inter-mingling, which makes it hard to understand what's going on.

Currently, our directory layout looks this:

bin (build scripts ) doc ( API documentation ) img ( images used in documentation ) projects ( project source data ) release ( gitignored, release-able builds ) schema ( project.json schema) src

I propose we rearrange our project to look like this:

bin (build scripts ) doc ( API documentation )

You may notice that there isn't a space for the Three.JS code anymore. That's because as a part of this proposal, I think we should refactor the Javascript library into a module that can be built with webpack, allowing us to distribute the library as a single, minified Javascript file that has Three.JS, and any other dependencies automatically rolled in. This would also allow us to write tests for the library. The build step for webpack can easily be added to the make_release script, so the build process doesn't get any more complicated.

My hope is that, by more cleanly separating the different components of the project, and by more clearly labeling directories, it'll be easier for newcomers to understand what's going on, as well as easier for all of us to work on separate components without stepping on each others' toes.

I'd like to hear the others' feedback on this. If you like, I can create a separate branch in my fork with these changes so we can try it out.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/lanl/4DGB/issues/14, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABIX4AEP7VV53S7CA2SZB63TYWIHLANCNFSM5AWA3UKQ.

camtauxe commented 3 years ago

That sounds like a good plan, I'll figure something out for the project shema.

If Ricardo still has things to do, I don't think a code freeze will be necessary. So long as the changes are relatively minor, I can handle a few merge conflicts.

@ricardosida, I have a few branches from you in my repo: rsida, sida and test_gentk. Do any of these still need to be merged into master before continuing, or are we good-to-go?

camtauxe commented 3 years ago

I have branch where I have mostly finished this here. However, please don't merge it in yet, since I'm about to be gone for a week and a half and won't be able to help if things break. You can lift the code freeze and continue doing whatever you want on the master branch. I'll just play catch-up with this once I get back.

Still, I think it might be helpful to understand some of the changes so far. So I'll describe them here: Please note that I haven't touched any of the testing code yet, so currently they all fail since the files they need are in different places now. Building releases, running the server, and viewing them in a web browser does all work, however. Just the automated tests fail.

All the directories are re-arranged how I want them, with the code for the three distinct components separated. I've also re-arranged the layout of the created release directories in a way that, hopefully, will make more sense once it becomes time to integrate a production HTTP server (instead of using Flask's built-in testing server). Also, options such as port and host are now given as command-line options when you run gtkserver.py, and not when you run make_release.

The Javascript library is now built using webpack (this will happen for you when you run make_release). All the classes are as they were before, but I dropped the GTK prefix from their name and they are instead available as members of a global GTK object (i.e GTK.ArrowSegment instead of GTKArrowSegment). THREE.js and d3.js are installed through npm and are automatically rolled into the build by webpack. (webpack is also installed through npm btw). My hope is that this system gives us more freedom with the Javascript library and makes refactoring and feature development easier in the future.

Finally, I've updated the readme to reflect these workflow changes (as well as included a small readme that will go into each release directory).

dhrogers commented 3 years ago

Looks good. Hopefully the merge when you get back won't be too difficult!

camtauxe commented 3 years ago

Update: I'm back at work (albeit with very limited internet) and will get to work on merging this with all the changes from the last two weeks.

dhrogers commented 3 years ago

Great!

Welcome back :)

From: Cameron Tauxe @.***> Sent: Thursday, August 5, 2021 9:30:44 AM To: lanl/4DGB Cc: Rogers, David Honegger; Comment Subject: [EXTERNAL] Re: [lanl/4DGB] Proposal: Re-organizing directory layout (#14)

Update: I'm back at work (albeit with very limited internet) and will get to work on merging this with all the changes from the last two weeks.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/lanl/4DGB/issues/14#issuecomment-893553443, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABIX4AHGN4JVCAHVOOKJZSDT3KVCJANCNFSM5AWA3UKQ. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

camtauxe commented 3 years ago

Update: I've caught up with all the changes in master and my refactor branch is now based off the latest commit in master. It works the same way as the changes I described in my earlier comment, which is to say I still haven't tackled fixing all the tests. That will be next :)

dhrogers commented 3 years ago

Redesign looks great, @camtauxe! Closing this issue.