Excel-DNA Roadmap Discussion

govert commented 6 years ago

How should Excel-DNA evolve over the coming years?

I can imagine the following versions, with some features that may be progressively realised over the coming years.

version 0.35

[x] Improve build tasks - more reliable clean-up and debugger detection.
[x] Improve RTD and async QueueAsMacro reliability
[x] Improve install process of ExcelDna.AddIn NuGet package (now requires NuGet 2.5)
[x] Call UnhandledExceptionHandler for macros (ExcelCommands) too
[x] Change how ExcelDnaUtil.Application works in Protected View - try harder but don't cache

Released as soon as possible - end of May / early June 2018.

version 0.36

Increase IntelliSense flexibility
Improved unloading support
Improve build process (Excel-DNA itself, not user add-ins)

Some example issues:

[ ] #67 Allow complete unloading of add-in - implement DllCanUnloadNow()
[ ] #85 Store and return untruncated attribute descriptions
[ ] #49 Fix AppVeyor build

version 1.0

Final version to support .NET versions < 4.0 and Excel versions < 2007

Maybe by 2019?

version 1.1

First version with only .NET 4.0+ and Excel 2007+, preserving interface (existing add-ins should work / easily recompile)

Rework marshaling implementation
Add support for high-performance marshaling with Span, in-place parameters etc.
RTD marshaling improvements (prevent multiple result copies).

version 2.0

Add support for .NET Core, including pre-compiled add-ins with CoreRT (also allows wabasm support for running as add-ins on other platforms).

Add-In projects might look very different, but we should try to preserve behaviour of the code as far as possible.

Maybe in the early 2020s?

augustoproiete commented 6 years ago

Love it!

version 0.36

Improved unloading support

I'd say this is very high priority and would open the door for a better story at updating Excel-DNA powered add-ins without having to restart Excel, which would be a huge win!

I think if we get this working and are confident enough, it's so important that would be worth releasing it alone as v0.36 and leave whatever else is left for a v0.37+ release.

Possible suggestions for future versions, from v0.37 to v9000 😃

Some of these will never get done, some we are already working on, others might have a better fit as a separate project instead of core Excel-DNA, etc. but for discussion purposes below are some possible suggestions (and dreams) I can think of right now.

The themes and items are not sorted in any particular way... It's all just a brain dump:

Improve the experience of professional developers / teams

[ ] Change the lifecycle of add-ins so that we have a single and consistent entry-point (IExcelAddIn) and objects get instantiated in the right order - so that when an ExcelRibbon gets instantiated, our entry point already ran and set up everything the ribbon needs to work
[ ] Add support for Dependency Injection and Unit Testing and make them first-class citizens
[ ] Add support (and samples) for add-ins using MV*** patterns
[ ] Add support for async and await (e.g. IExcelAddInAsync with an AutoOpen that returns a Task)
[ ] Add support for App.config XTD transformations out-of-the-box (relates to #241, #282)
[ ] Add support for SourceLink to enable developers to step into Excel-DNA's code as they are debugging their project
[ ] Improve the API for ExcelDna.Registration and make it the default & preferred way for registering functions from File->New Project
[ ] Add support for mocking Excel as a whole ( 🦄 one can dream, right? )

Improve the add-in developer experience (all levels)

[ ] Add reliable support for add-in deactivation when Excel is being closed, with enough time to do clean-up, free resources, etc. to allow for graceful shutdown. Expose a CancellationToken developers can rely on to break/terminate gracefully any async tasks that are running (if any)
[ ] Create Roslyn analyzers to detect common issues in the project (e.g. Your class must be public for the macros to be registered with Excel)
[ ] Add support for creating Ribbons directly via .NET code, without having to write any XML
[ ] Add support for implementing ribbons specific for Excel 2017 / Excel 2010 / etc. and have Excel-DNA automatically load the right one based on the user's Excel version
[ ] Add TlbExp option to build targets
[ ] Convert ExcelDna pack into a build task

Improve the add-in end-user experience

[ ] Allow users not to have to think about launching the right add-in (32-bit/64-bit) and just do the right thing for them
[ ] Resolve the issue(s) that causes add-ins to get disabled sometimes and exceptions are thrown in the user's face
[ ] Have a way for developers to specify requirements / rules to run their add-ins, and (before running the add-in) check if the user running the add-in has the requirements and gracefully stop loading the add-in if we know it's not going to work (e.g. user doesn't have the required .NET version, required version of Windows, enough disk space, ...)
[ ] Add support for (automatically) embedding metadata (and code) in Excel files, so that when a user sends a spreadsheet to someone that does not have the add-in installed, the user (at least) gets a nice message saying that they need to install add-in X for the spreadsheet to work

Keep up with changes in the .NET ecosystem

[ ] Add support for the NuGet PackageReference format (a.k.a SDK-style projects) (#193, #188, #138)
[ ] Add support for .NET Standard (#175)
[ ] Add support for developing add-ins with Visual Studio Code

New functionality out-of-the-box

[ ] Implement a RemoteFunction feature similar to XLLoop providing a spec/protocol so that devs can develop functions in whatever language they want (in the back-end)
[ ] Implement an inter-process communication feature so that different add-ins can talk to each other and dynamically share functionality

Community / Open-source / Support

[ ] Finish the conversion of the website to GitHub Pages and launch it. Let our developers update the website themselves and send pull-requests
[ ] Implement a better documentation / wiki integrated with the GitHub Pages site above using DocFx and let users update the documentation themselves in-place directly on the GitHub web user interface, just like Microsoft is doing now:
[ ] Wholesale copy contents of valuable answers to common problems and/or guidance/recommendations (this for example) to the wiki implemented above to stop answering the same questions over and over. Aim to always answer questions with a link to a wiki article that either existed or was just created for the purpose of the response. "We have a finite number of keystrokes left in our hands before we die"
[ ] Retire the Excel-DNA Google Group and make StackOverflow the official channel for asking questions and getting support. SO is much better ranked on search engines making it easier to find answers to problems and decreases repeated questions being asked over and over like in the Goggle Group today
[ ] Increase the # of contributions/pull-requests from developers by leveraging more of GitHub's features
- [ ] Create issues for everything that gets added to the roadmap so that people can chip in
- [ ] Allow users to assign issues to themselves, so
- [ ] Use "Projects" to better visualize the roadmap on a board as well as a kanban board to better communicate current progress, what needs scoping, what's ready to start (by anyone), etc.
- [ ] Create tag existing issues as firsttimersonly, awesome-for-beginners, up-for-grabs, etc. to give our issues more visibility to the dev community and attract newcomers to contribute
[ ] Promote the use of Gitter or Slack channels for exchanging ideas of possible features, bug fixes, etc. in a more informal way and (sometimes) less asynchronous than GitHub issues
[ ] Ship pre-release versions of the NuGet packages on every pull-request that gets merged, automatically. e.g. Have the AppVeyor publish a pre-release package to a NuGet feed somewhere (e.g. via free/public MyGet automatically on every successful build. Let users try new features/fixes as soon as they are merged
[ ] Experiment listing big / important features on BountySource (or similar service) to try to get funding for building features faster. Some of the above are massive undertakings that are not feasible without proper (or at least some) funding

govert commented 6 years ago

Thanks @augustoproiete.

Let me try to put your list into a few buckets:

1.0 - a good feature to work on now, hopefully targeting 1.0 time frame ('final' version with current structure)
1.1 - a good feature to work on now, hopefully targeting 1.1 time frame (mostly compatible with the current project, extra features and
2.0 - something that might targeting a 2.0 time frame, where we might consider large changes to the project and add-in structure
Sample / Independent Project - means an orthogonal feature that does not need changes to the core project, and can be done based on the current project (i.e. based on the version 1.0 feature set)
Not sure - means I'm not yet convinced that this is a good idea, and we should discuss before any work is done.

Improve the experience of professional developers / teams

Change the lifecycle of add-ins so that we have a single and consistent entry-point (IExcelAddIn) and objects get instantiated in the right order - so that when an ExcelRibbon gets instantiated, our entry point already ran and set up everything the ribbon needs to work

Not sure / 2.0 - I still don't understand what you need here. We can restart a separate discussion for this.

Add support for Dependency Injection and Unit Testing and make them first-class citizens

Not sure / 2.0 - I'm open to some suggestions for improvement that will make this easier to bolt on externally, but definitely against any built-in support for DI.

Add support (and samples) for add-ins using MV*** patterns

Not sure / 2.0 - I'm open to some suggestions for improvement that will make this easier to bolt on externally, but definitely against any built-in support for these patterns.

Add support for async and await (e.g. IExcelDnaAsync with an AutoOpen that returns a Task)

Not sure - I don't understand what is needed here.

Add support for App.config XTD transformations out-of-the-box

Not sure - I don't know what this means, or what parts of Excel-DNA it would impact. Open to discussion.

Add support for SourceLink to enable developers to step into Excel-DNA's code as they are debugging their project

1.0 - I'd like this, but don't really know what it entails. Perhaps part of the "Improve build process (Excel-DNA itself, not user add-ins)" item I've listed above for version 0.36?

Improve the API for ExcelDna.Registration and make it the default & preferred way for registering functions from File->New Project

1.1 - I'm happy with this if we can make it compatible. It requires the .NET 4+ move envisioned as version 1.1.

Add support for mocking Excel as a whole ( 🦄one can dream, right? )

Sample / Independent Project - probably you'd split this into a new 'host' that can load and deal with any native .xll add-in (not too hard), and something that mocks the Excel object model (not hard but lots of work - maybe ClosedXML is a start as it already has a C# implementation of much of the object model.

Improve the add-in developer experience (all levels)

Add reliable support for add-in deactivation when Excel is being closed, with enough time to do clean-up, free resources, etc.

Sample / 1.0 - Is the notification you get when a COM add-in (e.g. ribbon) is disconnected good enough?

Expose a CancellationToken developers can rely on to break/terminate gracefully any async tasks that are running (if any)

Not sure / Samples - Not sure what the technical suggestion is here. For native async functions the Excel cancellation support is implemented like this. For RTD-based async functions the Task-based wrappers in the Registration extensions do this. Perhaps for QueueAsMacro this is something to consider.

Add support for implementing ribbons specific for Excel 2017 / Excel 2010 / etc. and have Excel-DNA automatically load the right one based on the user's Excel version

Not sure - This works already.

Add TlbExp option to build targets

1.0 - I'd like this

Convert ExcelDna pack into a build task

1.0 - I'd like this. Should keep separate ExcelDnaPack as an .exe too (it might reference and load the build task library with the 'real' implementation. Maybe add code signing to this too?

Create Roslyn analyzers to detect common issues in the project (e.g. Your class must be public for the macros to be registered with Excel)

Not sure - I don't know what this means, or what parts of Excel-DNA it would impact. Open to discussion.

Add support for creating Ribbons directly via code, without having to write any XML

Sample / Independent Project - Nothing stops this now. Even a visual designer might be nice to incorporate.

Improve the add-in end-user experience

Allow users not to have to think about launching the right add-in (32-bit/64-bit) and just do the right thing for them

Not sure / 2.0 - I don't have good technical plans for this. We can investigate again whether the magic DllExport feature of .NET will allow the right runtime to be picked automatically, but I doubt it. An alternative is to load a COM add-in that then does RegisterXLL, but the overall user experience is even worse. Another alternative is to make a small start-up .exe which asks the user which Excel to load stuff into, or starts a new Excel and loads the right add-in. No support in Excel-DNA needed for the last two suggestions.

Resolve the issue(s) that causes add-ins to get disabled sometimes and exceptions are thrown in the user's face

1.0 / 1.1 - I'd like this. One options is to wrap all ribbon calls ourselves by implementing an IDispatch wrapper. That allows us to catch any unhandled exceptions.

Add support for (automatically) embedding metadata (and code) in Excel files, so that when a user sends a spreadsheet to someone that does not have the add-in installed, the user (at least) gets a nice message saying that they need to install add-in X for the spreadsheet to work

Sample / Independent Project - I like this. Nothing stops this now as a little sample.

Have a way for developers to specify requirements / rules to run their add-ins, and (before running the add-in) check if the user running the add-in has the requirements and gracefully stop loading the add-in if we know it's not going to work (e.g. user doesn't have the required .NET version, required version of Windows, enough disk space, ...)

Sample / Independent Project - Note sure this fits into the core library.

Keep up with changes in the .NET ecosystem

Add support for the NuGet PackageReference format

1.1/2.0 - I'm not sure of the short-term benefits. Depends on the technical issues involved.

Add support for .NET Standard

1.0 / 1.1 / Not sure - AFAIK we can already consume .NET Standard libraries. I don't know if there is some issue with assembly redirection - we should check this. I'm not sure what the implication or value is of making ExcelDna.Integration target .NET Standard.

Add support for developing add-ins with Visual Studio Code

Not Sure / Sample / Independent Project - What does this mean?

New functionality out-of-the-box

Implement a RemoteFunction feature similar to XLLoop providing a spec/protocol so that devs can develop functions in whatever language they want (in the back-end)

Sample / Independent Project - I like this, but not as part of the core project. Nothing stops this now and I've seen a few implementation with Excel-DNA.

Implement an inter-process communication feature so that different add-ins can talk to each other and dynamically share functionality

Not Sure - I'm not sure I'd encourage this.

Community / Open-source / Support

Finish the conversion of the website to GitHub Pages and launch it. Let our developers to update the website themselves and send pull-requests

Not Sure - My hesitation here has to do with the split between what is 'mine' and what is the 'open-source' project. I need a little bit more time to think about this.

Implement a better documentation / wiki integrated with the GitHub Pages site above using DocFx and let users update the documentation themselves in-place directly on the GitHub web user interface, just like Microsoft is doing now:

Yes - I'd like this. I think we should start a documentation site that is separate from the GitHub Pages site (e.g. at docs.excel-dna.net).

Wholesale copy contents of valuable answers to common problems and/or guidance/recommendations (this for example) to the wiki implemented above to stop answering the same questions over and over. Aim to always answer questions with a link to a wiki article that either existed or was just created for the purpose of the response. "We have a finite number of keystrokes left in our hands before we die"

Yes - I agree.

Retire the Excel-DNA Google Group and make StackOverflow the official channel for asking questions and getting support. SO is much better ranked on search engines making it easier to find answers to problems and decreases repeated questions being asked over and over like in the Goggle Group today

No way - I won't consider this at the moment, even though I answer questions on both. The Google group is much friendlier and usable than StackOverflow for this kind of thing..

Increase the # of contributions/pull-requests from developers by leveraging more of GitHub's features

Create issues for everything that gets added to the roadmap so that people can chip in

Allow users to assign issues to themselves, so

Use "Projects" to better visualize the roadmap on a board as well as a kanban board to better communicate current progress, what needs scoping, what's ready to start (by anyone), etc.

Create tag existing issues as firsttimersonly, awesome-for-beginners, up-for-grabs, etc. to give our issues more visibility to the dev community and attract newcomers to contribute

Yes - I agree and I'm trying.

Promote the use of Gitter or Slack channels for exchanging ideas of possible features, bug fixes, etc. in a more informal way and (sometimes) less asynchronous than GitHub issues

Not sure - I prefer asynchronous email communication, and especially the searchable Google group list. I feel like important discussions just get lost in chat rooms.

Ship pre-release versions of the NuGet packages on every pull-request that gets merged, automatically. e.g. Have the AppVeyor publish a pre-release package to a NuGet feed somewhere (e.g. via free/public [MyGet] automatically on every successful build. Let users try new features/fixes as soon as they are merged

0.36 - I'd like this - it falls under the "Improve build process (Excel-DNA itself, not user add-ins)" item I've listed above for version 0.36.

Experiment listing big / important features on BountySource (or similar service) to try to get funding for building features faster. Some of the above are massive undertakings that are not feasible without proper (or at least some) funding

Not sure - We can try, but I can't imagine crowdfunding working for this.

augustoproiete commented 6 years ago

Thanks @govert. I'll get back to you on your comments above. Unfortunately it seems you were replying to my post at the same time I was editing it and adding more stuff, so there are some items in the different themes that I'm assuming you didn't see because you didn't comment on them. When you have a chance, could you see if you spot anything else you want to comment on?

govert commented 6 years ago

@augustoproiete OK - I've updated my responses according to your edits.

I'm not sure how to structure the discussion - not ready for every item to be an individual issue though. We'll see if anyone else chimes in over time.

augustoproiete commented 6 years ago

@govert Great, thanks. Not sure how to structure the discussion either other than creating individual issues and/or opening a channel on Gitter or Slack to trash things out.

I'll wait a little to see if anyone else wants to give input - their own wish lists and/or comments to anything we've outlined above.

/cc @edparcell, @mndrake, @Ron-Ldn, @konne, @imanushin, @kchen0723, @Wildbiftek, @rlethbridge, @alainbryden, @sizzles, @phrrngtn, @autobotsassemble, @rosecodym, @masariello, @andrewkittredge, @dreamershl, @tomikiss, @hell-racer, @Mad-Lynx, @markns, @MisesEnForce, @WestXu, @vincentbeaulieu, @PGsHub, @ittegrat, @buybackoff, @akothoor, @davidandersen, @ian-cole-mcdavid, @ch-hristov, @rvnlord, @dominhtri1995, @chrish29, @WestXu, @ishiikurisu, @ghostjiao, @Rand-Random, @jamaynor, @sharpe5, @tarshinsn, @fandrei, @bpierce88, @pmunin, @ricardok1, @sanchugh, @chelh, @darul75, @robodude666,@stripwax, @Zupnik, @gigi81, @zmarwa, @limkinZero, @zmohamedfakier, @drdbkarron | - apologies if I forgot anyone. Tried to tag all previous contributors to this repo. Please, do chime in even if your name is not in this list

phrrngtn commented 6 years ago

I am happy to see this discussion and I am definitely interested in contributing some suggestions. The one which has been nagging me for years is a 'modern web browser in a CTP' with some thought given to making bidirectional communication between the browser and Excel tractable.

I like the sound of a bunch of @augustoproiete suggestions but it will take me some time to go through them. I will also dust off some suggestions that I have sent directly to Govert over the years and see how they fit in (and if they are still relevant).

Excel DNA is an excellent product and I think that a great many useful things can be built around and on top of it.

alainbryden commented 6 years ago

Great discussion. I may have some ideas to contribute in the near future (ideas to support installers, auto-updates, managing user settings, etc).

This one caught my eye and I felt the need to comment on it:

Add support for (automatically) embedding metadata (and code) in Excel files, so that when a user sends a spreadsheet to someone that does not have the add-in installed, the user (at least) gets a nice message saying that they need to install add-in X for the spreadsheet to work.

@augustoproiete We usually handle this by having a cell in a prominent place containing a formula like: =IFERROR(IF(NOW()+N(ARe.SomeUDF()),"Thanks for using ARe"),"You need the add-in! are.com")

If you wanted to automatically do that to all your formulas, you could probably have a (pretty invasive) event handler that, On Save:

Bulk search/replaces all formulas containing UDFs from your add-in to wrap them with something like the above,
Then saves the workbook,
Then undoes the change so the user can be none the wiser.

You would then pair that with a workbook-open event handler that also does step 3.

konne commented 6 years ago

Hi, I really like this discussion but not the form. Be onest after was finished reading, I already lost the most of the requested features. I would like to have more an issue related discussion. @govert @augustoproiete would it be okay for you to create for each suggestion an issue. Please add Releases / Labels and just add them. (like Help Wanted,...) An then just use features like thumb up/down,... do get an feedback of the users.

I agree to the most of your mentioned feature, and I think they just have to be ordered and we have to try to find contributors.

The nice think on my side is that I have now more R&D Power on my Excel Plugin so that I can even take one of my employes and let him develop some parts. The build process was in the last time mostly changed from caiproiete so that I didn't touch the msbuild task, but that is something that we can add very fast.

govert commented 6 years ago

@konne I agree - I'll wait a little while for any more feedback here, then split into a wiki page and separate issues.

Any particular features that you are interesting in seeing or working on?

konne commented 6 years ago

a real installer / helper for installing of the plugins multi language for autocompletion / help context

govert commented 6 years ago

@konne Have you tried the WiXInstaller?

Can you explain the multi-language thing?

augustoproiete commented 6 years ago

@phrrngtn: Love the "modern web browser in a CTP" idea. It would be really useful. In the past I ran into the same issue that you did with CefSharp back in 2016, which to this day is still not resolved... Would be great to put together a team to either fix this and make it work, and/or explore other alternatives such as headless Chrome that didn't exist back then.

@alainbryden: I really like the simplicity of using an Excel formula to show the message about the add-in, and I've been using something along those lines coupled with worksheet properties that store the add-in name and version inside the worksheet, and inject some VBA that then gets used. Would like to explore this area more.

konne commented 6 years ago

Have you tried the WiXInstaller? yes, I already build some Intallers, I like to have a general Template with a cool WixInstaller hostest in /Excel-DNA/

Can you explain the multi-language thing? You can add for UDFs Names / Descriptions,.. this should be possible to have a out of the box multi language support.

andrewkittredge commented 6 years ago

Me and another person at Calcbench (.com) have spent thousands of hours working on our ExcelDNA based add-in in the past three years. Neither of us are Windows desktop developers by training or inclination. Our Add-in is one of our marquee features and we are told that it compares favorably with the likes of Bloomberg.

@Konne We started with the WIX template and eventually got it to work. WIX is the typical MSFT layer-cake that takes three or four tries but eventually does most of what you want. We do not really have installation problems anymore.

@phrrngtn It took me a month but I got Chromium in a CTP work last year. I discussed our approach @ https://groups.google.com/forum/#!searchin/exceldna/chrome%7Csort:date/exceldna/PflxmqgvMkw/Gh5YAY1SAgAJ, which you commented on. We now have pipes going in both directions, from the Excel process to the Chromium process, and the other way. The chromium feature is stable now, though it does not work at a particular bulge-bracket investment bank where I really wish it did, I have no idea why.

The biggest problem with our add-in now is that it does not work on Mac. This isn't an issue for most of our professional users but many of our users are students, who love their Macs. I realize that Excel-DNA will probably be Windows/COM only until the end of time.

I played with the Javascript add-in approach that Microsoft is talking about right now. It is terrible. We will continue to watch it but they have a habit of losing interest in these things.

phrrngtn commented 6 years ago

@andrewkittredge I was unable to get this going in my environment (I strongly suspect an application virtualization mechanism in use at the time). I have to confess that I was not sure what needed to be done in order to get it up and running.

@augustoproiete I work in Reinsurance and will try and look you up the next time I am in Bermuda.

augustoproiete commented 6 years ago

@augustoproiete I work in Reinsurance and will try and look you up the next time I am in Bermuda.

@phrrngtn Small world! Yes, please do. I'll send you an e-mail with my contact info.

augustoproiete commented 6 years ago

Idea: What about something like FeatureUpvote (similar to UserVoice, except it's free for us*) to gauge votes from everyone here (we could send a link to the Google Groups too) to help us prioritize the discussions/items we want to scope out which ultimately can influence what goes in the roadmap and when?

https://exceldna.featureupvote.com

(*) Steve McLeod, founder of FeatureUpvote, was nice enough to give us a free plan after I signed up for a 30-day trial to experiment with it for Excel-DNA.

From: Steve McLeod Date: Tue, May 22, 2018 at 8:09 PM Subject: Re: Feature Upvote for open-source project - Excel-DNA To: C. Augusto Proiete

Hi Augusto,

It is our pleasure to help Excel-DNA with tracking feature requests.

Your account is now on our "freebie" plan and ready to use.

Regards,

Steve McLeod Founder, Feature Upvote

phrrngtn commented 5 years ago

wrt browser in a CTP, I saw Webview component based on Edge: https://blogs.windows.com/msedgedev/2018/05/09/modern-webview-winforms-wpf-apps/

Apparently, this is part of Windows 10 1803. It would be great if this 'just worked' with ExcelDNA.

SeeSpotRun commented 5 years ago

Add support for developing add-ins with Visual Studio Code

Not Sure / Sample / Independent Project - What does this mean?

I assume this means providing tools and templates (if required), and a tutorial / worked example, of how to develop and ExcelDNA Add-in using the Visual Studio Code editor, instead of having to use the full Visual Studio IDE.