Not only did TENEX have nothing to do with TOPS-10, but also it wasn't really a new design in its own right, being merely a rewrite of the SDS 940 system for the PDP-10. The 940 system, as well as the hardware enhancements that turned the 930 into the 940, came from Project Genie at UC Berkeley. The hardware enhancements were taken over by SDS (later XDS) to become the 940, though I'm not sure if SDS distributed a version of the Project Genie OS or not. There were certainly service bureaus running it, though, since I recall using one of them in '68.
The 940 OS (mostly written by Peter Deutsch) was based on paging, viewed memory primarily as a cache for drum, had jobs with multiple forks which could optionally share pages, used the topmost fork of the job for the "shell", and had command and filename completion. Sound familiar? Although the 940 only had a 16KW (24 bits) address space (64KW physical), the OS made page switching fairly convenient (and reasonably efficient) so that programs could "juggle their way" around that with overlays.
BBN had developed a popular LISP implementation that ran on the 940, and was getting squeezed by the limited address space. They wanted to move to another machine with more addressing, and the PDP-10 seemed like a good candidate, but it lacked paging (this was years before the KI). So they decided to build their own pager for the KA10, with some rather complex features designed to put some of the 940 OS's mechanisms into hardware. They also came up with the ugliest "fixes" for the KA10's difficulties with recoverable memory protect violations anywhere (and a step backwards from the 940). I know of at least three cases of fixing this "right", before DEC started getting it right on the KI and later.
TENEX was really not much more than a port of the 940 OS to the PDP-10, except for changing some things that didn't scale well to larger address spaces and storage devices. I can recall seeing TENEX code that looked literally like a line-by-line translation of 940 code, right down to using only three registers (like the 940's A, B, and X), and with half the instructions being MOVEs and MOVEMs since you can't keep much in three registers. They did, however, use a stack and PUSHJ/POPJ instead of JSR/JRST. :-) The exception model for system calls was as ugly as their hardware, instead of adopting an ITS-like PCLSR approach - another step backwards from the 940 system, which in essence had simplified PCLSR.
When ARPA got onto their kick of wanting to standardize on a single PDP-10 OS for all ARPA sites, TENEX was the only available choice that supported paging, which of course was a highly desirable feature. Thus, TENEX was selected as the "ARPA standard" OS on that basis alone. However, BBN had initially designed TENEX solely as a vehicle to support BBN LISP. It was designed almost in a vacuum, drawing only from the 940 system while ignoring the substantial benefits of some of the features in ITS and WAITS. Needels to say, there was a lot of kicking and screaming over the prospect of giving up important features just so that some bureaucrats in DC could see the same commands no matter what ARPAnet site they connected to, and eventually ARPA relented on the "TENEX mandate". DEC's willingness to take over TENEX and turn it into TOPS-20 may have played a part in this; I'm not sure.
One of the things DEC did with TOPS-20 was to junk the TENEX filesystem, which was widely known to be the most unreliable of all PDP-10 filesystems, and start over. It's still not as robust as the TOPS-10 filesystem, but it's a lot better than TENEX. The flaky TENEX filesystem was probably also a 940 legacy, but the 940 only used it on the drum. Long-term storage was either on magtape (and it had an elaborate means for allowing magtape files to be overwritten in place), or on disk partitions accessed purely through a user program called KDF (Kludge Disk Files). Restoring the drum filesystem from tape was a frequent occurrence.
Fred Wright wrote: