The first two lines of each CSV were not exactly the same. So it would cause the 2nd header to add to the file each time it was merged. Another issue was that when the files would merge, the last line of the previous CSV would have the first line of the new CSV being merged and append to the rest of that line.
So, I had to add a new blank line to the end of each CSV that way the next one being merged would start on a fresh line, to avoid having more than 108 columns.
Current Solution
I extracted the first 2 lines as headers. This also meant that Tail needed to start from the 3rd line of each CSV file to grab the unique data from there onward. This alone fixed duplications, but not the other error. The 108 characters expected limit also was a part of the redundant headers issue, except now it was unique data coming from the remaining 3 CSVs after the first CSV was merged.
Notes
I checked the current merged CSV file versus the one used in my version. It seems that all data matches up, except maybe the order which was also tested. Thus, maybe the version I compared it to was an older version since it had fewer lines of data than the version I was using.
Root Cause of Redundant Headers
The first two lines of each CSV were not exactly the same. So it would cause the 2nd header to add to the file each time it was merged. Another issue was that when the files would merge, the last line of the previous CSV would have the first line of the new CSV being merged and append to the rest of that line.
So, I had to add a new blank line to the end of each CSV that way the next one being merged would start on a fresh line, to avoid having more than 108 columns.
Current Solution
I extracted the first 2 lines as headers. This also meant that Tail needed to start from the 3rd line of each CSV file to grab the unique data from there onward. This alone fixed duplications, but not the other error. The 108 characters expected limit also was a part of the redundant headers issue, except now it was unique data coming from the remaining 3 CSVs after the first CSV was merged.
Notes
I checked the current merged CSV file versus the one used in my version. It seems that all data matches up, except maybe the order which was also tested. Thus, maybe the version I compared it to was an older version since it had fewer lines of data than the version I was using.