search

joyfullservice / msaccess-vcs-addin

Synchronize your Access Forms, Macros, Modules, Queries, Reports, and more with a version control system.
Other
203 stars 40 forks source link

Potential improvement: Importing and exporting recent changes only #81

Closed mmzmusicnotes closed 3 years ago

mmzmusicnotes commented 3 years ago

Hi @joyfullservice,

I've done some work recently on the original branch of this project which I think you might find valuable. In short, I implemented the following:

Essentially, the goal is to make the process as fast as possible by only importing/exporting objects which are likely to have changed. Even with fast save, some of our databases take upwards of 10 minutes to export, and nearly a minute subsequently. These changes make it take less than half of that; combined with your faster save, I think it would cut things down to be nearly instant.

Once that's working, it would make sense to have a button or command that can just do a quick update of that kind, rather than going all the way through the export/build UI (which is lovely, but takes many clicks). That button-press would then become a normal part of the git workflow - export, commit, push; fetch, pull, import.

I did all this before I knew about your approach, otherwise, I would have started from your code instead, as it seems to do quite a bit more, and more quickly. We use the plugin through the immediate window; it would be nice to have a way to access this that was integrated more completely into the VBA editor (rather than either typing into the immediate window or having to go to Addins and open the UI, configure it, etc).

If you are interested in integrating this work into your project, I would be happy to help you get this working in your add-in.

joyfullservice commented 3 years ago

@mmzmusicnotes - Thanks for reaching out! On the performance side, 10 minutes seems really long, even for a very complex database... Is that with the joyfullservice version of the add-in? I have a very complex Access ADP project that exports about a thousand objects. That takes just under a minute for a full export, and under four seconds for subsequent exports. (And most of that time is dealing with SQL objects, not database components.)

image

I actually used to have a toolbar in the VBA IDE, but I moved away from that because I preferred the approach of a demand-loaded add-in that I could use on any database instead of needing to add code to each project to load the VCS tools.

On the workflow side, I personally use GitHub Desktop alongside Access in my development work. When I am ready to make a commit, I simply run the export from the add-in, then switch over to the GitHub Desktop to review my changes, add the commit message and push the changes.

image

It sounds like your version is much more tightly tied to git itself, whereas mine is looking at file modification dates, which also allows you to use other file-based version control systems like CVS, SVN, TFS, etc...

That being said, if there is something specific that you feel would really improve the process, I would be happy to take that into consideration. šŸ˜„

mmzmusicnotes commented 3 years ago

@joyfullservice Thanks for the quick response!

On the performance side, 10 minutes seems really long, even for a very complex database... Is that with the joyfullservice version of the add-in?

It is. Here's my initial export:

image

and subsequent:

image

Import is similar. Clearly, my DB is much larger than yours (and arguably, larger than anyone's ever should be)!

I actually used to have a toolbar in the VBA IDE, but I moved away from that because I preferred the approach of a demand-loaded add-in that I could use on any database instead of needing to add code to each project to load the VCS tools.

Is that code somewhere in this repo? It would be enormously helpful to us, I think...

Or, does the add-in have a way to add that functionality? I'd be happy to help get it working if so.

On the workflow side, I personally use GitHub Desktop alongside Access in my development work. When I am ready to make a commit, I simply run the export from the add-in, then switch over to the GitHub Desktop to review my changes, add the commit message and push the changes.

Our workflow is somewhat similar, but substitute Sourcetree for GitHub. The big advantage to integrating into git is that you can go both back and forward in time. For instance, suppose that I revert to an earlier commit. My code will determine the delta of changed files between the last commit I imported and that earlier reversion, and import only those files. Unless I misunderstand what you are doing (or your audit trail is much more robust than I realized), I don't think that can be done with modification dates?

If you desired it, this could be available as an option - a method of deciding what to import. That way, for other source control systems you can still use file modification dates.

joyfullservice commented 3 years ago

Thank you for the additional background information, and the screenshots. Wow, I have never seen that many queries in a single database. šŸ˜„ It makes sense why you would be wanting to improve the export performance.

As to the code for the VBA IDE toolbar, absolutely! You can browse the related code here. Specifically, you are probably wanting to see clsVbeMenu and modMenuHandler. If you would find something like this helpful in your development cycle, I would be happy to integrate a pull request into the core. (I would probably also suggest an option where you could choose whether or not to load the VBE menu.)

The integration you are describing with git sounds intriguing... I would be interested in reviewing a pull request on how you would suggest integrating this into the project. As you have probably discovered, I have completely restructured the source files for this project, moving towards a more object-oriented class-based approach that can more easily be refactored and extended going forward.

I love the idea of being able to combine efforts with other skilled developers on this project. The more people we have behind this project and supporting it going forward, the more all of us will benefit in the long run. šŸ˜„

mmzmusicnotes commented 3 years ago

Wow, I have never seen that many queries in a single database. šŸ˜„ It makes sense why you would be wanting to improve the export performance.

And that's nothing - we have one DB with over 25,000 queries! (There are some good and some not-so-good reasons why things are this way, but regardless, we need to capture the current state before we can go forward.)

We are also working on a continuous integration method whereby we use automated tools (e. g. TeamCity) and an autoexec script to rebuild the database from source on a build server for each development increment. That isn't up and running yet, but when it is, it will probably use a version of this tool.

As to the code for the VBA IDE toolbar, absolutely! You can browse the related code here.

Thank you! Yes, I think this would be helpful for us, especially if I integrated menu items for our "export changes", "import changes" and "[export/import] all [object type]" options.

The integration you are describing with git sounds intriguing... I would be interested in reviewing a pull request on how you would suggest integrating this into the project.

I'd like to give that a shot. It may be a couple of weeks before I get a chance, as we've just finished this current pass of work on the system (using the structure of the original version of the plugin), but after that I can hopefully try to dig in and see how we can integrate it.

joyfullservice commented 3 years ago

And that's nothing - we have one DB with over 25,000 queries! (There are some good and some not-so-good reasons why things are this way, but regardless, we need to capture the current state before we can go forward.)

I totally understand! It is nice to have a current state before diving into major development.

We are also working on a continuous integration method whereby we use automated tools (e. g. TeamCity) and an autoexec script to rebuild the database from source on a build server for each development increment. That isn't up and running yet, but when it is, it will probably use a version of this tool.

You might be interested in #51 where a couple of us were discussing some build automation using this add-in. I think this is something that would be of interest to several other Access developers as well.

I'd like to give that a shot. It may be a couple of weeks before I get a chance, as we've just finished this current pass of work on the system (using the structure of the original version of the plugin), but after that I can hopefully try to dig in and see how we can integrate it.

Sounds great! I think you will find the code structure fairly self-explanatory, but feel free to reach out if you get stuck on something. I appreciate your interest and collaboration on this project. šŸ˜„ Hopefully the many hours of cleanup and restructuring I have spent on it will give you a jump-start on the development side as you work out issues with very large and complex databases.

joyfullservice commented 3 years ago

@mmzmusicnotes - As of the latest release (v3.2.1) I have implemented some additional performance tuning and logging. I am guessing that you will see noticeable improvement when exporting your databases. šŸ˜„ I am definitely interested in your idea of a "Merge Build" where the changed files are integrated into an existing database based on the date of a head commit rather than a complete rebuild each time.

mmzmusicnotes commented 3 years ago

Hi @joyfullservice - thank you for your work on this!

Unfortunately, the new version is crashing on this line:

Log.Add "[" & cCategory.Count & "]" & IIf(Options.ShowDebug, " " & cCategory.Category & " processed.", vbNullString) which is line 115 of modImportExport. It does seem to be going faster, though!

I am continuing to work on our git implementation. Once I have it in a relatively stable place, I will put it up on GitHub as a starting point, and then we can discuss integrating those changes here.

joyfullservice commented 3 years ago

Is the error on the export or the import? Sounds like maybe a quick fix...

mmzmusicnotes commented 3 years ago

Sorry, I should have specified - on the export.

joyfullservice commented 3 years ago

Could you post a screenshot of the error message? Also, if you click Debug and use F8 to step through the code when it hits the error, can you see what the issue is?

mmzmusicnotes commented 3 years ago

Ran it again, and it's now working -- not sure what the issue was. Here's the output:

image

Definitely a savings.

joyfullservice commented 3 years ago

Nice! At the bottom of the log file, you should see some performance reports... I would be interested in seeing how those come out for your database... šŸ˜„

mmzmusicnotes commented 3 years ago

Here's what I've got:

-------------------------------------
        PERFORMANCE REPORTS
-------------------------------------
Object Type         Count     Seconds
-------------------------------------
themes              1         0.01
VB project          1         0.02
vbe references      12        0.05
imex specs          20        1.08
db properties       50        0.17
tables              813       55.07
queries             1938      481.20
forms               321       10.24
modules             76        2.17
reports             58        2.19
doc properties      6         0.04
nav pane groups     1         0.02
-------------------------------------
TOTALS:             3297      552.25

-------------------------------------
Operations          Count     Seconds
-------------------------------------
Console Updates     11        0.34
Extract Theme       1         0.00
Clear Orphaned      6         19.48
Increment Progress  799       1.76
Delete File         5244      3.25
Unicode Check       5595      4.57
Write to Disk       5595      6.40
Save Table SQL      813       16.24
App.ExportXML()     296       1.04
App.SaveAsText()    2393      263.13
Read File           2375      0.58
Sanitize File       2317      1.56
Save Query SQL      1938      107.64
Read File DevMode   58        0.03
Other Operations              150.25

If we can improve the performance of query/table export, that would be the biggest bang for the buck from my perspective.

joyfullservice commented 3 years ago

Thanks! That is helpful to see. As you may have seen, the two tables are basically two perspectives of the same export. The top table is by object type, and the bottom table is by process. The totals are roughly similar, but will have some expected variation based on the timing precision and other overhead details.

Based on my performance tuning development last week, here is my analysis:

image

Of this, about 70% is internal processes that we really can't speed up any further. (Such as Access actually serializing the object and writing it to a file.) I am encouraged to see that we are getting some really nice performance, even on a large and complex database.

Going forward, the biggest performance gains will obviously be in refining the selective export/import of objects.

P.S. I would also be curious to see the performance reports on a subsequent export using Fast Save. I am thinking that you should see some significant improvements there as well.

mmzmusicnotes commented 3 years ago

P.S. I would also be curious to see the performance reports on a subsequent export using Fast Save. I am thinking that you should see some significant improvements there as well.

Subsequent fast save, as you suspect, is much faster:

image

For workflows where most work is edited in Access, this is more than sufficient, I think, provided you don't have to export too frequently. We like to edit in VS code, though, where possible - it has many features the Access editor doesn't have, such as changing all references, searching globally for all references, and so on. We import into Access (using my git-based ImportChanges which is quite speedy) and then export any changes made to resolve debugging issues.

One thing I wonder is whether Application.SaveAsText is the only way for us to go for all objects. For instance, there is DoCmd.TransferText: https://docs.microsoft.com/en-us/office/vba/api/access.docmd.transfertext

or even using a query-execution approach: https://stackoverflow.com/questions/17272945/exporting-a-query-result-into-a-text-file-using-vba-ms-access

This of course is referring specifically to tables and queries, which are the long-tail in my databases.

Without knowing what Application.SaveAsText does internally, I am unsure whether any of these would be faster. (At a guess, TransferText could be, it seems to me, but a query-based approach seems unlikely to be.) I'm wondering if you know much more about any of these?

joyfullservice commented 3 years ago

@mmzmusicnotes - Thanks for the additional background on your workflow process. I will look forward to learning how you are interacting with Git and determining the changed files and such. Ultimately I believe this is going to be the missing link to fast and efficient merging of changes into a working database.

In terms of raw speed, the undocumented SaveAsText is pretty fast. The performance report you attached above indicates that files were being generated and written at a speed of just over a tenth of a second per object. I don't know if this is to a network drive or mechanical drive, but on my older laptop with an SSD, I am getting 0.04 seconds per object for a local export of 295 objects. Reduce that down with a Git integration to say, 10 objects that actually changed, and I think we could be running exports in under 2 seconds.

In regards to other functions to export database objects, I am familiar with those approaches you mentioned. They are for exporting the data stored in a database, as compared to the object definitions themselves. Based on my experience with this add-on and others like it, the SaveAsText function seems to be the very best approach that I am aware of for the purpose of serializing a database design for version control.

mmzmusicnotes commented 3 years ago

That makes sense - thank you.

I'm going to speak with my colleagues today about getting some of our git integration back out into a repo for you to take a look at. Not sure what form it'll take, but hopefully will be able to put it up soon. Thanks again for your help!

A9G-Data-Droid commented 3 years ago

@mmzmusicnotes, I'm just popping in to say that the RubberDuck may make working in the VBE more attractive than VSCode:

https://rubberduckvba.com/

Their ultimate goal is to replace the code window in the VBE with AvalonEdit but that is a long term goal. For now, it has some amazing refactoring and code inspection tools that I use every day.

mmzmusicnotes commented 3 years ago

@A9G-Data-Droid Hey, thanks for stopping by. Yes, we've tried Rubberduck - it tries to chew through our code and pretty much dies. It's not usable, just much too slow to do anything, and crashes frequently. Doubtless it would be less painful in a DB of smaller size... though if you have any ideas on optimizations that would make it usable, I'd be interested.

@joyfullservice since I am here, an update - I got the go-ahead to get my code uploaded to Github. We are working on our CI pipeline and that's revealed a few bugs in our implementation I'd like to get resolved. Regardless of that, though, I will put up what I have before the end of the month. Just hoping a little extra time will further improve the state of things.

mmzmusicnotes commented 3 years ago

@joyfullservice to get us started, I went ahead and threw up the current state of things here:

https://github.com/mmzmusicnotes/msaccess-vcsintegration-gitsupport

I'm sorry that this does not have the history from the original repo, and is thus not a true fork. I moved it into our internal source control, and there was not a clean way for me to set that up. I spent a little time trying to rebase it off the original repo, but that turned out to be a pain - and as I said, this is just a proof-of-concept.

The parts to look at are the VCS_GitOperations module and the related changes to VCS_References and VCS_ImportExport - there is commit history to show you how those changed. I would start with the ImportChanges and ExportChanges functions in ImportExport to get an idea of what's going on.

I think if we wanted to integrate this work here, we would want to be much more circumspect about architecture. I threw this together and it is fairly fragile (as the original plugin was). Obviously, the git stuff could all be contained in a class that handles git operations. We'd also want to do a better job of managing tags (right now, users have to manually create the access-vcs-last-imported-commit tag when they set up the VCS plugin).

Finally, there's some challenges with export - when you export your changes, it may include changes that were last imported from git, because export has to go by date of modification, and you must always export all modules, because I have not found a reliable way to ascertain when a module was modified. I also don't yet have a good way to notify git that a module was renamed or delete within Access itself - but as I mentioned, this is not a super high priority for us as we mostly work in VS Code/the filesystem, and then import the changes into Access.

The UI is also worth discussing. If this is presented as a setting or option, then it could change the import/export behavior accordingly. It might be better, though, to have it as an entirely different set of operations - e. g. a "Sync with git" button that exports any changes made in Access, and imports any changes made in git, and updates the tagged commit. You could even provide a list of commits and choose visually which one to import changes starting from (by moving the tag).

I hope this is helpful!

joyfullservice commented 3 years ago

@mmzmusicnotes - Thanks for uploading a copy of your project! This was exactly what I needed to be able to see how you were interacting with git. I like your technique of reading the StdOut from the shell commands. I have never used that before, but it looks like a very effective approach.

I agree with your suggestion of using a dedicated class for the git interaction. Perhaps clsGitSync. I am not too worried about some of the items not having modified dates. Most of them are going to be lightweight items that we can export to a temp file and compare with the repository file to identify any changes.

I think the bigger question is going to be how we work out the overall workflow and development cycle. We need to consider files that may have changed in the following areas:

Obviously the most important thing here is that a user does not inadvertently overwrite their changes and lose work. Any time changes are overwritten from one source to another, there needs to be an intentional decision on the part of the user as to which changes are preserved.

After some consideration here, I would like to propose the following approach for managing the changes, then outline what this looks like from a workflow perspective in some different development scenarios. This would be the behavior with the git integration functionality turned on.

A guiding principle here is that we are using GitHub Desktop, SourceTree, or other git management tool to handle commits, pushes and pulls to the remote repository.

Export Source:

Merge Build:

Use Cases:

Primary Development in Microsoft Access Periodic exports to source gives the developer a chance to review the changes and commit them to git. Changes made since the last merge build would continue to be exported (but probably not flagged as changed in git) until the next merge build is performed.

Collaborative Development in Microsoft Access Differences in the source files and database objects would need to be resolved at the source file level, then a merge build performed before source could be exported.

Primary Development in Source Files Periodic merge builds would keep the Access objects up to date with the latest changes made in the source files.

After reviewing this in more detail, I would suggest that we take the following plan of action towards integrating this into the core.

  1. First, let's establish the complete specification of how we want this to work, especially in reference to dealing with change conflicts, to make the process as clear and intuitive as possible for the users. (I went ahead and added a page to the wiki to get us started using what I described in this post.)
  2. With the specification completed to satisfactorily meet the various usage requirements, we can move forward with the code updates to implement the specification into the software.

While this takes a little more planning and thought at the beginning, I believe it will give us the best quality product in the end, help minimize surprises and reduce the potential of developers losing work when something doesn't function as expected.

What do you think? Do you like this approach?

mmzmusicnotes commented 3 years ago

Apologies for the delay on looking at this - want to make sure I give it complete thoughts.

In general, I like all of the goals you've laid out, but I want to take an iterative approach in terms of getting the functionality into the plugin. More advanced work around resolving conflicts, etc., I think we should set a specific policy on to start, and then add more complex functionality after we've built the basics out if needed. For instance, if there's a change conflict - the exported/imported file/object changed and the target file/object also changed after the last import/export date - I think for now we just alert the user and let them choose which version to keep (or to skip the file).

Likewise, I appreciate the defensive approach as far as avoiding file overwrites, but I want to give the user the power to choose the "dangerous" option as long as they know they're doing it. For instance, in this scenario:

To keep from overwriting changes in the source files, an export can only be performed when there are no uncommitted changes in the source files folder, other than code exports (see above). Any existing changes to source files must be discarded, stashed or committed first.

I would say that an alert and the option to proceed would be preferable.

When exporting, with Fast Save on, only objects that have changed between the last export date, and the last commit date in the repository are exported. This should give us a very fast export.

This sounds like a great idea, similar to what I've done for import.

Certain types of changes are ignored when writing files, such as changes in the VCS version in .json header sections, and some properties that may frequently change.

Agreed; this can be achieved on git's end via .gitignore. In access we'll obviously have to do it explicitly. We should also ignore any local settings the add-in may save, as different users may want different things.

A couple of minor notes as well:

I am not too worried about some of the items not having modified dates. Most of them are going to be lightweight items that we can export to a temp file and compare with the repository file to identify any changes.

So long as we use git diff to do this, I agree.

Also, there's a unique issue I've run into with import/export that we should be aware of. If you import changes, then export immediately, Access will see the modification date as being when the import happened, and will thus flag those changes to be exported. We can get around this by updating the json on both import and export, as I think you intend?

Lastly, with regards to how we turn this functionality on and off - what were you thinking? This is a fairly big feature, so maybe we want some way of making it clear what you lose if you disable it?

joyfullservice commented 3 years ago

Thank you for the feedback and thoughtful replies! I feel like we are both on the same page, just working through the finer details as we move towards implementing this enhancement. I have added some replies to your comments below:

In general, I like all of the goals you've laid out, but I want to take an iterative approach in terms of getting the functionality into the plugin. More advanced work around resolving conflicts, etc., I think we should set a specific policy on to start, and then add more complex functionality after we've built the basics out if needed. For instance, if there's a change conflict - the exported/imported file/object changed and the target file/object also changed after the last import/export date - I think for now we just alert the user and let them choose which version to keep (or to skip the file).

Sure, that sounds fine to me. Practically speaking we could probably do a scan first, then present the user with a single question rather than individual message boxes for each file affected. (Just in case there are hundreds or thousands of changed files.) It starts to sounds a bit complicated when we present a list of files and manage what to do with each of them. (It makes me think of a SyncBack confirmation screen.) But like you said, if we start with the basics, we can build from there as needed.

Likewise, I appreciate the defensive approach as far as avoiding file overwrites, but I want to give the user the power to choose the "dangerous" option as long as they know they're doing it. For instance, in this scenario:

To keep from overwriting changes in the source files, an export can only be performed when there are no uncommitted changes in the source files folder, other than code exports (see above). Any existing changes to source files must be discarded, stashed or committed first.

I would say that an alert and the option to proceed would be preferable.

Good idea. We could add this as an option, kind of like you have with SSMS where you can allow more dangerous queries to run if you enable the option. We might have this off or set to prompt by default, but it could be set to overwrite.

Certain types of changes are ignored when writing files, such as changes in the VCS version in .json header sections, and some properties that may frequently change.

Agreed; this can be achieved on git's end via .gitignore. In access we'll obviously have to do it explicitly. We should also ignore any local settings the add-in may save, as different users may want different things.

I am not sure I was quite following how this would be done in .gitignore... I know you can ignore whole files, but not easily ignore specific lines or parts of files.

The way my plugin does it for the json files is that it compares the content of the json file and only rewrites the file if the content section changed. See the source code here.

image

The VCS version can be important in the future if the structure of the file changes between VCS versions, and we need to transform the source content before importing. But we don't really need to update the version numbers in every json file every time we upgrade VCS. (That was the thought process behind this approach.)

Do you have some specific examples of local things that we would not want saved in VCS that are currently being saved? I have functionality for localizing relative paths, and also some handling of the printer options to help minimize this type of issue. The VCS options are saved with the project in the export folder, so other users working on the same project should be utilizing the same options and getting the same output.

Also, there's a unique issue I've run into with import/export that we should be aware of. If you import changes, then export immediately, Access will see the modification date as being when the import happened, and will thus flag those changes to be exported. We can get around this by updating the json on both import and export, as I think you intend?

Yes, this additional state catalog will allow us to more accurately determine when changes happen outside the export/import processes. I found some interesting posts regarding how git determines which files have changed, but git is obviously only comparing the source file to the git head version, and doesn't know anything about the modified date of an internal database component.

Lastly, with regards to how we turn this functionality on and off - what were you thinking? This is a fairly big feature, so maybe we want some way of making it clear what you lose if you disable it?

Yes, initially I was looking at having this option available, but off by default. When you check the box for Advanced Git Integration, it would make a new tab visible that would contain additional options relating to this feature. (Similar to how I did the hidden pages for print settings and encryption.)

Initially, the add-in would continue to function using the file-date based approach for Fast Save, but as the git integration matures, this may eventually become the default option, with the other still available as a fallback for those using other systems for version control.

joyfullservice commented 3 years ago

@mmzmusicnotes - I have created a new branch and began some initial implementation with the git integration. You can follow the changes and contribute on the git-integration branch. Since this involves some more significant refactoring, I thought it would be best to do the main development in the feature branch, then once it is working pretty well, we can merge it in with the master branch. I plan to pick this up again on Monday, and hope to make some good progress on it next week. šŸ˜„

@Indigo744 - You might be interested to follow this development as well, and potentially contribute if you are interested... When finished, it should significantly improve the performance of exporting and importing changes with git repositories.

mmzmusicnotes commented 3 years ago

@joyfullservice That sounds great. I will have fuller responses to your comments on Monday as well - just wanted to add one other thing I forgot to mention: It would be nice, for my team's uses, if we could have access to the VCS window or some shortcuts to export and merge changes in the VBA editor itself.

I think that can be a separate issue - in fact, I would encourage us starting Monday to chunk this work up into several issues under a Git integration "epic" of sorts, so we can each work on portions - but would you be willing to have that in the plugin if I provided it/included it as something optional?

joyfullservice commented 3 years ago

@mmzmusicnotes - Sure! I would be happy to include an option for a custom VBE menu. I can totally understand how that would be useful in your workflow.

I like your idea of breaking this out into chunks so we can better collaborate on the project. I have never used the GitHub Projects feature, but it looks like it might work nicely for something like this. I just set up a basic project for us to use as we work through the integration.

Indigo744 commented 3 years ago

@joyfullservice thank you for pinging me. I was following this thread and was interested in the initial improvements. And by the way I want to thank you and @mmzmusicnotes for the work!

I'm less interested in the advanced git integration, but that said I'll continue to follow the development and contribute if an opportunity arise. Good luck with your endeavour!

A9G-Data-Droid commented 3 years ago

On VCS Version: Instead of storing the version in every file exported, would it make sense to only output the VCS version in one place. I believe that it should be tracked in source control and not ignored but it shouldn't update every file to track this information. I know there could be problems with files that are exported from mixed versions but backwards compatibility will become critical as you embark on this journey. Changes will have to be planned carefully and tested to make sure it works. You could start with a version and say that you support every version from that one forwards.

joyfullservice commented 3 years ago

@A9G-Data-Droid - Thanks for chiming in! I appreciate your input.

On VCS Version: Instead of storing the version in every file exported, would it make sense to only output the VCS version in one place....I know there could be problems with files that are exported from mixed versions but backwards compatibility will become critical as you embark on this journey.

Thinking about this more, perhaps a better approach would be to make the distinction between the version of the VCS software, and the version of the file format. The VCS version represents which features have been added and which bugs have been fixed. The file format version represents how the file contents are organized, and would only change when we make a (breaking) change in how the data is stored.

Taking this approach, the file format version would very rarely change, and when it does, we would fill out the Upgrade function to transform the loaded data into the new structure.

image

The name of this property is important because we don't want any confusion about what it means. It is not the VCS add-in version, it is not the version of the database or the object in the database, it is not the version in git, or the version of the source class. Does File Format sound like a clear name, or would there be a better term for this?

Perhaps we don't even need this element at all until we have a breaking change from the present version. This could be passed as an optional parameter to the WriteJsonFile function with a default of 0.0 (technically 0, but with the syntactical hint to use 1.0, 1.1, etc...) If the format is > 0, then the line would be included in the export file. This seems to be the simplest and cleanest implementation while still fully backwards compatible.

The ultimate goal is that you could drag and drop source files from other projects, exported at different times from various versions of VCS, into a new project, build from source, and have everything load in correctly, regardless of which version was used to export the source.

A9G-Data-Droid commented 3 years ago

@joyfullservice That's a great solution. I would call it "Export File Format" because it's specific to the Export/Import process. Import versions can then cite compatibility with certain Export File Format Versions. There could even be a test method that keeps one of each version example file and certifies compatibility with each Export Version.

mmzmusicnotes commented 3 years ago

Hi all. Here are the issues I would recommend we create as thin slices for this project.

The following I'd call "MVP", in the sense that we couldn't call the feature working without them:

and then these I'd call nice-to-haves which can be done subsequently:

How does this sound as a starting point? @joyfullservice , if it works for you, and you are comfortable giving me the appropriate permissions, I can try to create these and hook them up to the project board. (Or I can create them, and you can do the project setup.)

joyfullservice commented 3 years ago

@mmzmusicnotes - Sounds great! I think you should have permissions to edit the project now, so feel free to set up these items.

mmzmusicnotes commented 3 years ago

@joyfullservice I don't seem to be able to add or edit cards, or to update the projects or labels for issues, unfortunately. I am creating the issues now, though, starting with #100.

joyfullservice commented 3 years ago

@mmzmusicnotes - Thanks! It looks like the invite is still pending... Maybe see if the email landed in your spam folder?

I have been making some good progress today in some of the initial integration. I will plan to push out an update soon so you can see how things are coming together.

mmzmusicnotes commented 3 years ago

Ahh yes, it fell into a different folder. I have what I need now. Will have each of those issues created tonight, and obviously we can refine their requirements as I go. Happy Thanksgiving!

mmzmusicnotes commented 3 years ago

@joyfullservice I've created all the issues I listed above and fleshed them out as best I can. I also set up mvp and post-mvp labels so we can prioritize individual issues. Finally, I set up an automatic trigger for the project so that issues added to it go to "to-do" to be prioritized, and I added a "review" column for work awaiting approval/merge.

I know this issue chain is getting chatty - if you would prefer to email me privately to coordinate issues and other things that fall outside of the scope of this particular thread, my email is on my profile.

joyfullservice commented 3 years ago

@mmzmusicnotes - Sounds great! Thanks for your help in organizing this project. I am thinking that we can probably close this issue since we have completed the original objective of determining that yes, we would like to move forward with a tighter git integration and the related performance improvements. šŸ˜„

joyfullservice commented 3 years ago

For future reference, I am moving the wiki page of specifications to this issue. (I don't have a way to hide the wiki page, and this isn't a feature that is on the roadmap for further development at this time.) If we decide to pick this up again in the future, we can move this back to a wiki page with the other documentation pages.

NOTE: New Feature Under Development

If you are using this add-in with a .git repository like GitHub or GitLab, you may be interested in enabling a deeper git integration. Out of the box, this add-in uses the file modified date/time to determine when to export an object when the Fast Save option is turned on. The Advanced git Integration allows the add-in to interact with the actual git commits to determine which files have changed. Furthermore, it allows a Merge Build to integrate changes directly into a working database instead of needing to rebuild the entire application from source each time. See #81 for more discussion on the development of this feature.

Because this tighter integration can result in files being overwritten or replaced, it is very important that you understand how to use this functionality works.

SPECIFICATION

This outlines the intended functionality of the software in regards to the Export and Import of source files with the Advanced Git Integration option turned on.

A guiding principle here is that we are using GitHub Desktop, SourceTree, or other git management tool to handle commits, pushes and pulls to the remote repository.

Export Source:

Merge Build:

Use Cases:

Primary Development in Microsoft Access Periodic exports to source gives the developer a chance to review the changes and commit them to git. Changes made since the last merge build would continue to be exported (but probably not flagged as changed in git) until the next merge build is performed.

Collaborative Development in Microsoft Access Differences in the source files and database objects would need to be resolved at the source file level, then a merge build performed before source could be exported.

Primary Development in Source Files Periodic merge builds would keep the Access objects up to date with the latest changes made in the source files.

This PRELIMINARY specification was last updated November 18, 2020