Recently, I watched a video by YouTuber ThePrimeTime which details a dev’s frustrations with business culture. Prime is an ex-entrepreneur who currently works in software development at Netflix. His views in this video have been criticised for being jaded by FAANG business cultures he has worked in. I personally don’t feel this way. Although there is some truth to the developer’s (mataroa’s) article, I think it misses the root causes of many issues raised.
I have mixed feelings about the article in question. Some areas I agreed with and others I disagreed with. However I did want to address one particular concern / statement:
I work on a platform that cost my organization an eye-watering sum of money to produce, over the span of two years, and the engineers responsible for it elected to use spreadsheets to control the infrastructure, so we now have a spreadsheet with 400 separate worksheets that powers but one part of this whole shambling mess.
I’m speculating here, but I’d imagine that the business/SMEs (Subject Matter Experts) are using VBA to some capacity to control their 400 worksheet collection. So this begs the question…
Why do people use VBA?
In order to answer this question, we must first look at another question - who actually uses VBA in the first place? In 2021 I ran a poll on /r/vba where I asked redditors why they code in VBA.
From these data, we can clearly see that the majority of people who use VBA do so mainly because they have no other choice. Many organisations run their entire business processes with Excel, and when a little bit of automation is required VBA is usually #1 on the list.
The versatility of VBA
In the business I currently work for, in the engineering division, we have access to a variety of technologies (automation platforms):
OnCloud - Power Platform (PowerApps, Power BI, PowerAutomate (non-premium only))
SandboxedServer - ArcGIS (ArcPy)
SandboxedServer - MapInfo (MapBasic)
SandboxedServer - InfoWorks ICM (Ruby)
SandboxedCloud - ArcGIS Online
We also have a number of databases controlled by IT:
D1. OnPrem - Geospatial database
D2. OnCloud - Geospatial mirror
D3. OnPrem - SAP database
D4. OnCloud - SAP BW4HANA partial mirror
D5. OnPrem - Telemetry platform
D6. OnPrem - Sharepoint
D7. OnCloud - Sharepoint Online
D8. OnCloud - EDM Telemetry platform
D9. OnCloud - Large mirror database
D10. OnPrem - LotusNotes database
D11. OnPrem - IBM BPM database
D12. OnPrem - File System
D13. OnPrem - Hydraulic Model Information
D1-D13 databases are summarised in the table below listing what types of data are stored in which systems, the importance of the data stored in each database, and whether the database is essentially a replica of OnPrem information:
Data Type
D1
D2
D3
D4
D5
D6
D7
D8
D9
D10
D11
D12
D13
Customer Issues
X
X
X
X
X
Asset Data
X
X
X
X
X
X
X
X
X
Telemetry Data
X
X
Risk Data
X
X
X
X
X
X
Financial Data
X
X
X
Misc Data
X
X
X
X
X
X
Note: D6 would be C tier if it weren’t for the fact we continue to store a business critical spreadsheet on Sharepoint OnPrem for compatibility reasons. See the control of VBA for details.
And the data’s importance / on cloud replication:
Data Type
D1
D2
D3
D4
D5
D6
D7
D8
D9
D10
D11
D12
D13
Data Importance
S
S
S
S
S
S/C
A
B
S
S
S
S
A
Replica of
N/A
D1
N/A
~D3
N/A
N/A
N/A
N/A
~D4
N/A
N/A
N/A
N/A
Note: Online replicas are de facto replicas in terms of data’s importance, although the reason we need to connect to them is diminished
Finally, let’s examine how our Automation Platforms link to our Data Platforms. Links shown in the diagram are where the automation platform can access the data from the various data platforms/sources:
Note: Some of the links from VBA to OnCloud services are based on my attempts alone. There is no doubt in my mind that VBA can interface with SAP BW4HANA and our other cloud services, I just haven’t figured out the authentication requirements and protocols yet
Here’s where you might start to see an issue. Looks like the only automation platforms which can connect to all the data sources we need is VBA and Powershell. Power BI Desktop has been introduced in our business but doesn’t hit all the platforms which VBA does, and even if it did Power BI cannot be used for process automation where-as VBA can, so what’s the point making the switch? Users who do use Power BI to target these other datasets usually generate CSVs of this other data and store these in cloud sharepoint system, but what generates those CSVs? VBA.
Now, we’d love to use a higher level language in our organisation to handle this business automation. However, every request for a high level language to be installed across the team/business e.g. Python / Ruby / Node / Rust etc. has been rejected by CyberSecurity in favour of technologies like PowerAutomate, PowerApps which as you can see above barely touch any of the data we need. It is supposedly “Against the technology strategic vision of the company” to allow “end-users” access to high level programming languages. Now even if the data access was there in our business, PowerPlatform would still be insufficient to perform the majority of our processes because the algorithms required are so complex that a PowerAutomate solutions would become infuriating to maintain and incomprehensible to even IT folks (e.g. See projection algorithms).
Ultimately the stand-out technologies for us are Powershell v3 (doesn’t even support class syntax and cannot install modules), and VBA, purely from a versatility standpoint. As a result of this ‘monopoly’ on technology I and others have spent hundreds of hours building open source VBA libraries which augment VBA promoting it to a reasonable language by modern standards.
The maintenance guarantee of VBA
D10 and D11 above are intimately linked. In 2000s many of our systems were built on top of IBM Lotus Notes databases. In 2019 Lotus Notes was acquired by HCL, and since then longevity of support has been wavering. Support will officially die in June 2024. As a result, since 2019, technology teams have been trying to migrate many of our systems to new technologies. The business spent an eye watering amount of money developing a system using IBM Business Process manager to supercede one of these Lotus Notes databases. The anticipation was that D11 would be backfilled with all the data from D10, once fully built, and D10 would be archived.
It’s now 2023:
We are 8 months away from official support dying.
Technology teams have thrown away their support contact for IBM BPM.
There is no replacement in sight for both IBM BPM and Lotus Notes databases.
IBM BPM solution is poorly maintained
IBM BPM solution has numerous issues and doesn’t function as needed
Solution has been shoehorned into IBM BPM, despite the platform not being fit for purpose
i.e. while IBM BPM does come with a REST API, this REST API is borderline useless to Technology teams and SMEs
Some REST calls use javascript encoded as strings
Others require html embedded in json embedded in xml
Database tables aren’t queried by name but by GUID.
There’s no documentation of which GUID relates to which table/process.
The data from D10 was never actually transferred to D11, meaning the business is now using 2 systems instead of 1.
D11 data model doesn’t really support the data in D10 either.
Technology teams appear to be ignoring waning support contracts.
This could be due to lack of resources, finances and/or priority.
SME’s use these tools on a daily basis, and ultimately it is SME’s who determine changes that need to be made to the system. If SMEs use VBA, they can control and maintain the system as needed. They have a maintenance guarantee, something that should be said for IT systems too, but can’t be.
The control of VBA
In a recent project we are building a new all encompassing IT system to supercede a business critical spreadsheet. This would ultimately demote D6 to C tier importance. The spec for this system was initially simple - Give us a NodeJS server with a MySQL database. Use React for the UI. Give admins/SMEs (subject matter experts) access to the codebase with access to git for code control. IT and SMEs will collaborate to build the system.
Technology teams demanded that admins/SMEs will not have access to the code.
Technology teams demanded that FrontEnd be built in Microsoft PowerApps, to comply with “Strategic Vision”.
Technology teams demanded that BackEnd be built in Microsoft Azure Pipelines, to comply with “Strategic Vision”.
Unfortunately, as an admin/SME with more development knowledge than many people in technology, these demands do not sit well with me:
Technology teams do not understand work in the teams thus do not understand business logic and calculations
Thus devs writing business logic is error prone.
Technology teams has frequently abandoned bespoke technology projects, leaving no resource to maintain and improve the system.
Collaboration with SMEs will ensure that at least 1 team maintains resource to maintain system.
SMEs need to ensure that they have confidence in what is produced.
How, without observing that the code doesn’t work for all edge cases?
Unit tests?
Perhaps, but without seeing the code how can we verify these unit tests exist? And are ran frequently*.
SMEs improve and maintain the existing legacy system, and have unparalleled knowledge of how systems interact.
Less knowledge can be shared in upskilling Technology teams where it is required.
SMEs need to ensure all data is transferred and represented correctly in new system.
SMEs unable to do this without backend access.
Ultimately, as long as code stays in VBA it is controlled by the SMEs and the business. Technology teams rarely relinquish control to business teams. SMEs can ensure that software is developed properly in a modular fashion and doesn’t end up as a cluster of barely working technologies loosely linked together.
The UX of VBA
This is a smaller one, because you can always make familiar feeling tools in any language, but there is something to be said about the fact that most engineers use spreadsheets in their day job, and how VBA is embedded within these spreadsheets and can improve on UX. Giving someone a foreign tool in a familiar environment, is often much more powerful to that user, than a foreign tool in a foreign environment.
Conclusion
In conclusion, yes, we (and many others in businesses) do choose to use spreadsheets (and VBA) for many tasks within our organisations, there are many reasons for this, including:
Poor alternatives provided by IT due to security concerns.
Poor connectivity of alternatives to source systems, usually because they are still WIP.
Faults in IT strategy which don’t account for certain use-cases.
Unwillingness to collaborate with SMEs due to security and maintenance concerns.
Lack of training for users/managers/SMEs in alternative systems.
Users/SMEs wanting some level of control over the business logic in these systems.
It’s the only viable technology which is available to everyone, as it’s part of Office.
This does not mean that we are at all blind to VBA’s weaknesses though:
There’s no doubt in my mind that there are some elements of truth to mataroa’s article. Sometimes management is poor, but more often than not I believe most people in organisations are trying to do the right thing, and are doing whatever they can with the tools that are available to them.
Why do people use VBA?
https://ift.tt/Ned4jra
Sancarn
Why do people use VBA?
Introduction
Recently, I watched a video by YouTuber ThePrimeTime which details a dev’s frustrations with business culture. Prime is an ex-entrepreneur who currently works in software development at Netflix. His views in this video have been criticised for being jaded by FAANG business cultures he has worked in. I personally don’t feel this way. Although there is some truth to the developer’s (
mataroa
’s) article, I think it misses the root causes of many issues raised.I have mixed feelings about the article in question. Some areas I agreed with and others I disagreed with. However I did want to address one particular concern / statement:
I’m speculating here, but I’d imagine that the business/SMEs (Subject Matter Experts) are using VBA to some capacity to control their 400 worksheet collection. So this begs the question…
Why do people use VBA?
In order to answer this question, we must first look at another question - who actually uses VBA in the first place? In 2021 I ran a poll on /r/vba where I asked redditors why they code in VBA.
From these data, we can clearly see that the majority of people who use VBA do so mainly because they have no other choice. Many organisations run their entire business processes with Excel, and when a little bit of automation is required VBA is usually #1 on the list.
The versatility of VBA
In the business I currently work for, in the engineering division, we have access to a variety of technologies (automation platforms):
Install-Module
)We also have a number of databases controlled by IT:
D1
-D13
databases are summarised in the table below listing what types of data are stored in which systems, the importance of the data stored in each database, and whether the database is essentially a replica of OnPrem information:And the data’s importance / on cloud replication:
Finally, let’s examine how our Automation Platforms link to our Data Platforms. Links shown in the diagram are where the automation platform can access the data from the various data platforms/sources:
Here’s where you might start to see an issue. Looks like the only automation platforms which can connect to all the data sources we need is
VBA
andPowershell
.Power BI Desktop
has been introduced in our business but doesn’t hit all the platforms whichVBA
does, and even if it didPower BI
cannot be used for process automation where-asVBA
can, so what’s the point making the switch? Users who do usePower BI
to target these other datasets usually generate CSVs of this other data and store these in cloud sharepoint system, but what generates those CSVs?VBA
.Now, we’d love to use a higher level language in our organisation to handle this business automation. However, every request for a high level language to be installed across the team/business e.g.
Python
/Ruby
/Node
/Rust
etc. has been rejected by CyberSecurity in favour of technologies likePowerAutomate
,PowerApps
which as you can see above barely touch any of the data we need. It is supposedly “Against the technology strategic vision of the company” to allow “end-users” access to high level programming languages. Now even if the data access was there in our business,PowerPlatform
would still be insufficient to perform the majority of our processes because the algorithms required are so complex that aPowerAutomate
solutions would become infuriating to maintain and incomprehensible to even IT folks (e.g. See projection algorithms).Ultimately the stand-out technologies for us are
Powershell v3
(doesn’t even support class syntax and cannot install modules), andVBA
, purely from a versatility standpoint. As a result of this ‘monopoly’ on technology I and others have spent hundreds of hours building open source VBA libraries which augmentVBA
promoting it to a reasonable language by modern standards.The maintenance guarantee of VBA
D10
andD11
above are intimately linked. In 2000s many of our systems were built on top of IBM Lotus Notes databases. In 2019 Lotus Notes was acquired by HCL, and since then longevity of support has been wavering. Support will officially die in June 2024. As a result, since 2019, technology teams have been trying to migrate many of our systems to new technologies. The business spent an eye watering amount of money developing a system using IBM Business Process manager to supercede one of these Lotus Notes databases. The anticipation was thatD11
would be backfilled with all the data fromD10
, once fully built, andD10
would be archived.It’s now 2023:
D10
was never actually transferred toD11
, meaning the business is now using 2 systems instead of 1.D11
data model doesn’t really support the data inD10
either.SME’s use these tools on a daily basis, and ultimately it is SME’s who determine changes that need to be made to the system. If SMEs use VBA, they can control and maintain the system as needed. They have a maintenance guarantee, something that should be said for IT systems too, but can’t be.
The control of VBA
In a recent project we are building a new all encompassing IT system to supercede a business critical spreadsheet. This would ultimately demote
D6
to C tier importance. The spec for this system was initially simple - Give us a NodeJS server with a MySQL database. Use React for the UI. Give admins/SMEs (subject matter experts) access to the codebase with access to git for code control. IT and SMEs will collaborate to build the system.Unfortunately, as an admin/SME with more development knowledge than many people in technology, these demands do not sit well with me:
Ultimately, as long as code stays in VBA it is controlled by the SMEs and the business. Technology teams rarely relinquish control to business teams. SMEs can ensure that software is developed properly in a modular fashion and doesn’t end up as a cluster of barely working technologies loosely linked together.
The UX of VBA
This is a smaller one, because you can always make familiar feeling tools in any language, but there is something to be said about the fact that most engineers use spreadsheets in their day job, and how VBA is embedded within these spreadsheets and can improve on UX. Giving someone a foreign tool in a familiar environment, is often much more powerful to that user, than a foreign tool in a foreign environment.
Conclusion
In conclusion, yes, we (and many others in businesses) do choose to use spreadsheets (and VBA) for many tasks within our organisations, there are many reasons for this, including:
This does not mean that we are at all blind to VBA’s weaknesses though:
There’s no doubt in my mind that there are some elements of truth to mataroa’s article. Sometimes management is poor, but more often than not I believe most people in organisations are trying to do the right thing, and are doing whatever they can with the tools that are available to them.
Authors
via VBA Articles
November 17, 2023 at 04:21PM