Automatic Script file transfer at start of Scroll mode

[mmehr2]

This requirement wasn't listed yet, but is essential. For testing up until now I have relied on manually exchanging scripts by email. This won't work for customers. When scrolling a new file from the Primary, its text must be sent to the Secondary first.

What happens after depends somewhat on our design. I chose a design that requires only one-way messages from P>S. This was from discussions with the ReadyCam customer about the primary usage scenario, where the Operator is at the central site with a copy of MSWR Primary, and at various remote studios, there is an automated equipment rack with a teleprompter running MSWR Secondary, with the Talent. They seemed to be intent on eliminating the need for an Operator there, so I have tried to work that out. (See issue #16 for reasons why this might not work.)

As far as Editing mode goes, the Talent will be in a voice conversation with the Operator during their Session. The Operator will make edits on the Primary as requested by the Talent, and several things could happen with those:

1. Talent waits and when done, Operator goes into and out of Scroll mode to send the script again (entire), or just stays in Scroll mode while the reading continues
2. Same, but only the changes just made are sent to the Secondary (save file transfer time, if needed)
3. Variation in which the edits appear as they are made on both ends of the link (saves any waiting time by the Talent, they can see the requested changes being made). Might save transfers entirely, except at session start.
4. In all cases, entering Scroll mode does NOT need to send the file again if no changes have been made since the last Scroll mode was done.

After some back of envelope calculations, I thought that the transfer time wouldn't be so much of an issue. But we should measure it in the test scenario and then see if methods 2-3 are justified for the extra development work they imply.

[mmehr2]

This may turn out to automatically solve the problem in issue #14 about cross-machine resolution. Eric and I were testing the same file on our separate machines, but had our rulers set manually (and possibly not to the exact same settings), which would explain the slight differences encountered.

If the ruler settings (PARAFORMAT2 especially) are part of the RTF file sent, then we don't need a separate out-of-band message to make them match, internal operations of the richedit view class would take care of that when the remote text was loaded in.

Other than testing after this feature is complete, I'm not sure how to tell if this would work. But the idea of solving issue #14 by implementing this is a real time-saver if it works!

[mmehr2]

The step 4 optimization should be easy enough. The MswDoc object can override the virtual SetModifiedFlag() function to call something in the gComm class when the file is changed or saved. The problem is that we want a separate state flag that is set when the file is modified BUT cleared when the changes are sent to the Secondary, NOT when the file is saved.

Plus, the gComm function needs to determine if the doc file being modified is the one we are currently editing, i.e., it needs to update its state when the user changes windows. Or maintain a set of flags for every open window in case we want to send that one.

Ideally, that info would be kept in the MswDoc class and retrieved by the gComm object when script transfer needed to occur. Or the Scroll Dialog would check this flag and prevent the call to SendFile() if it isn't needed. Needs some thought.

[mmehr2]

Work Tasks:

Incorporating changes from further comments: @1 involves finishing with an EditStream instead of full file load if we already have the file open @2 involves being smart about sending if there have been no changes since last send. @3 is about holding off scrolling until file transfer is completed and ready

• Implement SendFile() function for Primary ([CB] denotes background thread).

• [ ] -- @2 Immediately exit if determined that no changes since last send to this Secondary

• [x] -- Put the BufferTransfer object into PubnubComm, configure differently on Primary (encoding) vs Secondary (decoding)

• [x] -- Hook up the Richedit transfer callback using the PNC object as the dwCookie

• [x] -- SendBuffer will send a new remote command (F?) to tell Secondary the file name, how many blocks and size per block (respecting 32kb limit set by Pubnub), and start kFilesending mode.

• [ ] -- During kFileSending, all subsequent messages are queued and not published.

• [ ] -- [CB]Callback w OK response in kFilesending will encode the next block and send that to the remote. If no more to send, turn off kFilesending mode and immediately start sending any commands that may be in the queue.

• [ ] -- [CB] @3 Need a second special "command" to be placed at the end of the queue as kFilesending mode shuts off. This will trigger the OK signal to start scrolling (post the first WM_SCROLL) once all other commands are sent. [Phase 2 - this gets sent as part of the callback Response from the Secondary finishing loading the file or script text.]

• [ ] -- [CB] Rely on the publish retry test to keep this going without changing the state. If it ends in an non-OK condition, we want to send the Abort command (X?) to the remote and turn off kFilesending mode.

• [ ] -- In kFilesending, also check for UI Cancel command (Esc), and if detected, send abort command to the remote to finish up (X?) without displaying (discard buffer), and turn off kFilesending mode. (This originally comes in on the UI thread, so a critical section is needed for setting the Cancel flag.)

• Implement script reception protocol on Secondary

• [ ] -- Make sure Sec has an empty BufferTransfer object on login.

• [x] -- When receive F command, set up for block transfer, saving data (FN, N and S) and set kFileReceiving mode. Possibly use a different sub timeout. QUESTION? - do we open the new MswDoc and its MswView here?

• [ ] -- When receive X command in kFilereceiving mode, shut off kFilereceiving and delete the buffer. Cancel any file open UI also.

• [ ] -- When receiving transaction timeout error, just restart the wait.

• [ ] -- When receiving network error conditions, just go dormant, and leave the data intact? Or act like the file was received anyway (unless no data).

• [x] -- When receiving encoded strings in kFilereceiving mode, count them, decode and add to buffer.

• [x] -- When the last string has been added, kick off the code to add it to the UI and close kFileReceiving mode. This code (to be run on UI thread) includes

• [ ] --- (file not open) creating a new file of the right name in the current folder (for now), saving the buffer data there, and loading the file as if the user had pressed File Open for it. It is possible we just need to post the correct message to the parent window, or invent a new one. See if I can find Steve's code to do this in the libjingle version.

• [x] --- (@1 file already open) stream the buffer contents into the Richedit control directly and re-render the control.

• [x] --- @2+@3 Send the file-transfer complete Response message. This will clear the Unsent Changes flag in the Primary's MswDoc on proper completion. It will also trigger scrolling to start.

• [ ] - Any additional requirements that were thought of in Steve's libjingle code. I need to find the receiver. There was some loops about removing various kinds of RTF markers, I think.

• Testing will require asynchronous debugging techniques, so it may be tricky to get it all right and robust. Especially the UI Cancel sequence, and/or response to Network errors, including short disconnect hiccups. Allow time for this somewhere (Beta testing?).

• [x] - @2 also involves inventing a new flag (Unsent Changes) as an extension of the SetModifiedFlag() on the MswDoc object. Since it only tracks when the changes are made between File Save operations, we want to set a new flag when this one is set, but clear that one only when a Script Transfer completes successfully. This should be done in a way that doesn't trigger the File Save operation again when the flag is cleared.

[mmehr2]

I need to update the task list to include the requirement for Session Edit Changes optimization.

This is two things:

• When sending the file the first time, it needs to be opened fully. After it is already open and in use, the change content can be loaded using the Richedit StreamIn() call/callback. This is even in Ph.1 where the entire script is sent.

• Primary needs the Send Only If Changed mode, so if no changes have occurred SINCE LAST SEND, they don't have to be sent again.

• ENH: Allow an editing protocol to specify any additions, deletions or changes separately. Is this feasible? For simple text edits anyway? What about formatting? Global changes (font, flow, etc.)? This could take a lot of work to do, and is risky in that it may not be easily doable at all.

[mmehr2]

Reliability feature: Need for back-channel ACK to the script transfer phase.

TROUBLE: If a NetworkError occurs on any block transfer sub call or get at the Secondary, the naive implementation operating blindly in one direction has no feedback that anything is wrong and will continue to operate in Prompt mode until the voice conversation alerts the Operator to quit (Esc).

If we had a feedback ACK message from Secondary to Primary to indicate transfer OK or fail, then the Primary could implement the Cancel command on behalf of the user, give them a Message Box, and go back to try-again state.

The same would need to happen if ANY subscription failure caused us to miss one of these blocks (check sequence counts), or network just hiccuped in the middle, or any number of things. Whether or not the Secondary can tell the Primary directly, it should actually put some indication up on screen that something went wrong. A user Message Box seems appropriate here, but if there is no one operating the UI, then makes it more difficult.

QUESTION: What kind of operator usage at the Secondary will the Customers require? Is this a configuration setting (Unattended/lights-out vs. Attended operation)? Or maybe it's a UI flag that someone can turn on if they are present and monitoring the Secondary?

[mmehr2]

I was a little unclear about how to hold off scrolling until the file transfer was completed. It looks like an easy thing to modify the InitDialog() of the ScrollDialog class to remove the final send of the WM_SCROLL message, and post it from the PNComm callback when the final block of the transfer has been sent.

Ideally, we'd wait for a completion confirmation from the remote Secondary, but without two-way comm, this is the next best thing. It might be good though to do the 2-way, because there is also bound to be some delay on the Secondary to save the script file the first time, open the window, and load the richedit control with the text. So a final ReadyToScroll response would probably be best. We don't want to assume zero delay, just give them the best chance of being in sync.

This would also need the timeout handler to notify the UI that the Secondary isn't ready yet, we think. And a prominent message on the Secondary so the voice link could confirm.

UPDATE 4/20: No need to do this. The SendFile() command should be able to wait for the busy condition of the file transfer to complete before sending the next command (g2 - scroll on).

[mmehr2]

Using a FILE offset replacement for BYTE

It's clear now to me that a version using FILE* offsets for fseek() would bypass all memory issues and by only encoding 32k chunks when needed, distributes the activity across multiple calls to the background thread, for maximum efficiency of both memory usage and processor time.

The existing code is based on files and even though Pubnub has no direct file transfer (yet), it's best to stick with this philosophy and try not to load the script into memory a second time (the contents of the RichEdit control are the first).

See the reserve() function for reasons why in-memory might not be a great way to go.

The actual offsets are only size_t numbers and can be pre-calculated using / and % when the file size is known (at initForCoding() time).

It's good to have an understanding of the tradeoff here with EDITSTREAM (see CRTFMemFile for other issues such as removal of page breaks that would cause this streaming to fail). Does this go away on the receiving end if we use the FILE approach? Perhaps only if we don't load the control with EDITSTREAM... well this has all been covered I think in use with existing files inside MSW anyway.

[mmehr2]

Realization: The flag to keep track of NoChangesSinceLastSend needs to be an array of flags, one for each Secondary that could be sent to, cross-indexed by MswDoc object (or attached to each MswDoc). All to save the need for sending repeatedly. Perhaps this isn't as good a feature as originally thought.

Example scenarios for testing:

1. Open doc 1. Scroll it to Secondary 1. Repeat scroll (it shouldn't do the file transfer the 2nd time).
2. Open another doc 2. Scroll it to Secondary 1. It should go the 1st time, but repeats should not send.
3. Make changes to doc 2. Go back to doc 1. Scroll it to Sec 1, it should STILL not send the file. Scroll it to Sec 2, it should send the first time (new combination of Doc1+Sec2), then not send after that.
4. Go back to doc 2. Scroll it to Sec1 - it should send the first time, then not again until modified.

Similar testing of interleaved scrolling using at least 2 docs and 2 secondaries will fail if the implementation only keeps one flag per document, or per the app. Still, it seems to be overkill.

One flag per document, defined for scrolling to the current Secondary seems adequate for our needs. However, we need to make sure that the flag is SET every time we switch Secondary connections. This will force a send to the newly connected secondary "just in case". Otherwise we'd have to compare the version of the script on the remote (if any) with the one on the Primary to properly set the flag.