Closed Captainkirkdawson closed 7 years ago
Captainkirkdawson
commented
7 years ago Neither Doug nor I can see any reason for such an event that appears to require a deliberate removal of the folder. None of our scripts involve such an action. It is not affected by git actions so likely required either human intervention or hardware malfunction. Screenshot uploaded to contact and feedback have been lost. Will take colobus out of production for FR while completing recovery.
Captainkirkdawson
commented
7 years ago Log investigation shows that problem occurred after a passenger spawning error "Cannot checkout session because a spawning error occurred. ."
dougkdev
commented
7 years ago Apache logs narrow down the timeframe that public went missing to between 20:48:16 GMT and 20:52:15 GMT on 02 Jan 2017 (probably in the 30-second window between 20:50:47 and 20:51:21).
dougkdev
commented
7 years ago I don't see anything in the auth.log that would point to any sudo commands that would have caused a problem in that time frame.
dougkdev
commented
7 years ago What time was the passenger error?
Captainkirkdawson
commented
7 years ago Was a Red Herring
According to the passenger log it was OK at 2017-01-02 20:51:19 +0000 and bad at just before 2017-01-02 20:51:21 +0000 this is a solid 3 hr before I saw the issue and it clearly was nothing to do with the restart. It was already gone.
What happened in between was an update of a refinery page
App 31799 stdout: Completed 200 OK in 22ms (Views: 10.3ms | ActiveRecord: 1.2ms) App 31799 stdout: Started GET "/search_queries/new?search_id=586abd40f493fd7c37968d41" for 5.9.125.163 at 2017-01-02 20:51:19 +0000 App 31799 stdout: Processing by SearchQueriesController#new as HTML App 31799 stdout: Parameters: {"search_id"=>"586abd40f493fd7c37968d41"} App 31799 stdout: Rendered search_queries/_form.html.erb (106.8ms) App 31799 stdout: Rendered search_queries/new.html.erb within layouts/application (107.1ms) App 31799 stdout: Rendered layouts/_main_nav.html.erb (3.1ms) App 31799 stdout: Rendered layouts/_header.html.erb (0.1ms) App 31799 stdout: Rendered layouts/_footer.html.erb (0.6ms) App 31799 stdout: Completed 200 OK in 125ms (Views: 112.6ms | ActiveRecord: 1.3ms)
App 31799 stdout: Started POST "/search_queries" for 5.9.125.163 at 2017-01-02 20:51:20 +0000 App 31799 stdout: Processing by SearchQueriesController#create as HTML App 31799 stdout: Parameters: {"utf8"=>"✓", "authenticity_token"=>"WxA46x4jVwX2ra+kFt2DtVV53WiONfr0VxfUsn6oCzTaNa8ZjdYQWTpYIBQIMHHwqPI2uXUMDkSqQwdt+S8ewg==", "search_query"=>{"last_name"=>"Martin", "first_name"=>"Ernest", "start_year"=>"1917", "end_year"=>"1917", "chapman_codes"=>["", "STS"], "place_ids"=>["5583614ef493fdf0f200169b"], "record_type"=>"bu", "region"=>""}, "button"=>""} App 31799 stdout: FREEREG:SEARCH_HINT: place_ln_fn App 714 stdout: Started GET "/cms/information-for-transcribers/creating-a-transcription-file" for 5.9.125.163 at 2017-01-02 20:51:20 +0000 App 714 stdout: Processing by Refinery::PagesController#show as HTML App 714 stdout: Parameters: {"path"=>"information-for-transcribers/creating-a-transcription-file", "locale"=>:en} App 31162 stderr: App 31162 stderr: -- SANITIZED CONTENT WARNING -- App 31162 stderr: Refinery::Pages::SectionPresenter#wrap_content_in_tag App 31162 stderr: HTML attributes and/or elements content has been sanitized App 31162 stderr: App 31162 stderr: App 31162 stderr: App 31162 stderr:
There are several ways to create a transcription file, but for Windows users our bespoke program WinFreeREG is the best choice. For those on Mac or Linux, we recommend using a spreadsheet, as found in any office software.
App 31162 stderr: App 31162 stderr:A FreeReg file should be no larger than about 2,000 records. We strongly recommend that those records be all for the same Place, Church and Register type: that is, they should amount to a single Batch.
App 31162 stderr:Before you begin to work, you must ensure that you have the necessary <a href="/cms/information-for-transcribers/permission-to-transcribe">Permission to transcribe? for your source.
App 31162 stderr:The new WinFreeREG is now available! You simply use this link to download the WinFreeREG installer and then run it. WinFreeREG is written and maintained by Mike Fry, an experienced transcriber.
App 31162 stderr:There are many advantages to using WinFreeREG:
App 31162 stderr:database readywithout any of the extra work required by using other programs.
For details of how to use WinFreeREG, see the
program's Help menu. Context-sensitive help (using the F1 key) is available
on most input screens: some have a white on blue i
icon that you can
click for more information.
If you are not able to use WINFreeReg for any reason, then we recommend using a spreadsheet, especially one from the Open Source office suites: the first two are available for Windows, Mac OS X and Linux; they are also free of charge.
App 31162 stderr:A modern version of MS Excel can be used, but we do not recommend older versions as they have a bug in the way they save CSV files — the kind of file that must be uploaded to the database. There is a way around this bug, but the fix involves extra work, and may itself introduce a different problem.
App 31162 stderr:Other proprietary spreadsheets are available, but we have less collective experience with these.
App 31162 stderr:You don't need to know how to do anything complicated with the spreadsheet, just how to: Save, Save as, and set the format of cells in a column (field) to Text. It would also be very useful to know how to freeze the headings, so that they are always visible. In addition, we ask that you turn off all the AutoCorrect options — the details of why and how are in the box below.
App 31162 stderr:We recommend that you save your working file in the spreadsheet's own format (file ending). This will make it simpler to re-open the file and will preserve such things as column widths and font size for when you next work on your transcription.
App 31162 stderr: App 31162 stderr:AutoCorrect or Substitution is now a common feature in many applications: even the so-called plain text editors like Notepad or TextEdit are not so plain any more. The default (or out of the box) setting is to have the feature turned on: this is not a problem for many kinds of writing, but for a database project like FreeReg, there are lurking dangers.
App 31162 stderr:Many of the corrections applied by AutoCorrect are to do with the casing of letters and the common typing errors people make with words. AutoCorrect can ensure that all sentences start with a capital (upper case) letter; it can prevent you beginning a word with two capital letters; and it can change the word i to I. It can also correct typos like abotu to about, or acn to can. This is not a problem in itself, but for FreeReg work, you will want to be in full control of how words are spelt and cased. The main problem is that AutoCorrect changes some plain punctuation to their smart (fancy or typographers) versions.
App 31162 stderr:A good example of AutoCorrect in action with punctuation happens like this: you type a plain double quotation mark on your keyboard: it starts out looking like ", but as soon as you type the next thing, it changes magically to “ — the plain quote has been converted to a fancy or smart quote. Again, this is not a problem for most kinds of writing (indeed, it is useful for a party invitations or a novel), but for FreeReg it is a problem.
App 31162 stderr: [31m-As well as changing " to “ or ”, other common substitutions are the single quotation mark (and apostrophe), ' to ‘ or ’; ellipses, ... (three characters) to … (one character); and dashes, - or -- to —
[0m App 31162 stderr: [32m+As well as changing " to “ or ”, other common substitutions are the single quotation mark (and apostrophe), ' to ‘ or ’; ellipses, ... (three characters) to … (one character); and dashes, - or -- to —
[0m App 31162 stderr:The exact form of the smart punctuation will vary from font to font, but there is less difference when comparing the plain forms. For the quotation marks, the plain forms will be vertical, with left and right quotes looking the same. (If you are having trouble seeing the difference, try zooming in: see your browser's View menu.)
App 31162 stderr: App 31162 stderr:If smart punctuation is used instead of plain, then a search for a surname like O'Grady will not find all the possible matches: the O'Grady entries will be found, but not O’Grady. The visual difference between the two apostrophes is small, but the effect on a search could be huge.
App 31162 stderr:Similarly, most abbreviations of a forename will be found automatically in a search, but any that include a smart apostrophe will not be found. So Tho's will be found in a search for Thomas, for example, but not Tho’s.
App 31162 stderr: App 31162 stderr:If your file has been saved using one character set, but the character set in the Headers is not the same, then any smart punctuation will not be understood by the database and you will see groups of characters like these in your uploaded batch: … “ � ‘ ’.
App 31162 stderr:This is most likely to happen when files are passed on to someone else for editing or Uploading: the remedy is to turn off all the AutoCorrect options, then re-type the problem punctuation. Finally, make sure that you put the correct character set in cell 1F of the Headers (see the <a href="/cms/information-for-transcribers/file-headers">File Headers page for details).
App 31162 stderr: App 31162 stderr:Start with a blank spreadsheet open: this will prevent you seeing any options that do not apply to spreadsheets. From the menu, select Tools > AutoCorrect Options. You will get a dialog box with four parts: Replace, Exceptions, Options, Custom Quotes or Localised Options. (Note that OpenOffice, for example, has a fifth part, Word Completion, which you will see if you do not have a blank spreadsheet open: you can ignore it.)
[0m App 31162 stderr: [32m+Start with a blank spreadsheet open: this will prevent you seeing any options that do not apply to spreadsheets. From the menu, select Tools > AutoCorrect Options. You will get a dialog box with four parts: Replace, Exceptions, Options, Custom Quotes or Localised Options. (Note that OpenOffice, for example, has a fifth part, Word Completion, which you will see if you do not have a blank spreadsheet open: you can ignore it.)
[0m App 31162 stderr:The third and fourth parts are the important ones: the first and second contain the details of how some correction options are applied — we are going to turn those options off.
App 31162 stderr:Part three (Options) has a list of possible corrections: simply de-select all of these.
App 31162 stderr:Part four (Custom Quotes or Localised Options) deals with quotation marks: un-check both Replace boxes. Un-check anything else that is offered here for writing in English.
App 31162 stderr:The CSV file, described below, that you will Upload or Replace on the server needs to be named according to the following pattern. The first 8 letters of the name are:
App 31162 stderr:For smaller registers of around two thousand records or less in total, these eight letters are all you need. However, if you are going to be transcribing a larger register of more than two thousand records, then you will need to append a number to each filename to distinguish each part of the transcription.
App 31162 stderr:For example, a baptism register of about six thousand records for a (fictitious) Place with code ABC in the West Riding of Yorkshire will need three files: WRYABCBA1, WRYABCBA2, and WRYABCBA3.
App 31162 stderr:If the register is large enough to need more than nine files to complete, then consider using two digits to number your files: this will result in a set of batches that sorts in a more natural order in batch listings. This is because batches are sorted in so-called dictionary order, so that WRYABCBA2, for instance, would be listed after WRYABCBA12, whereas WRYABCBA02 would be listed before it.
App 31162 stderr: App 31162 stderr:There is a final step which should be done in your spreadsheet application: adding the information that lets the database understand your transcription and know who created it. The instructions are on the File headers page. If you have any problems with this step, your Syndicate Coordinator will be pleased to help you.
App 31162 stderr: App 31162 stderr:When you have finished entering data, the next step is to save your file in CSV format. Just before doing so, delete the row of column labels, if you haven't already deleted them as part of adding the file Headers.
App 31162 stderr:With your spreadsheet open, use File, Save As to
bring up a dialog box. Choose Comma-separated values as the format,
probably from a drop-down list. You may be asked to confirm (or choose)
commas as the field separators
(or field delimiters
), and
double-quotes as the text delimiters
: agree to both. It is not
necessary to agree to all fields being quoted, but it will do no
harm.
A CSV file just a simple text file, with a .csv or .CSV extension (ending). Each data field (cell) is separated from the next by a comma, and each record is on its own line. For a baptism register, opened in a text editor such as Notepad or TextEdit (not a word processor), the beginning of the first two lines after the Headers might look something like this:
App 31162 stderr:GLS,Ruardean,All Saints PR,,,06 Jun
1786,Elizabeth,F, …
[0m
App 31162 stderr: [32m+ GLS,Ruardean,All Saints PR,,,06 Jun
1786,Elizabeth,F, …
[0m
App 31162 stderr: GLS,Ruardean,All Saints PR,,,20 Jun
1786,James,M, …
App 31162 stderr: Notice that some of the fields are empty in this example, with literally nothing between the commas: there are no unnecessary spaces.
App 31162 stderr: App 31162 stderr:This test is not necessary for users of Apache OpenOffice, LibreOffice or NeoOffice — all three pass.
App 31162 stderr:If this is your first transcription file with MS Excel (pre-2011) or another program not listed above, then this simple check will tell you whether or not it is suitable for FreeReg: in particular, some of the older versions of Excel have a bug in the way they save CSV files that makes them unsuitable.
App 31162 stderr: App 31162 stderr:The check is simply this:
App 31162 stderr: App 31162 stderr:Not sure? Some examples may help. A baptism entry with data in the final field — in this case, the Notes field — will end something like this, with no final comma:
App 31162 stderr: App 31162 stderr:… ,WARD,FORBES,"23, High St",,Mr Saml
Ward
App 31162 stderr:
(Note also that one of the fields has double quotes around the contents: this is because the contents — 23, High St — has a comma in it. Fields without a comma are not quoted.)
App 31162 stderr: App 31162 stderr:Lines without data in the final field, should end something like these:
App 31162 stderr: App 31162 stderr: [31m-A line without content in the
final field should end with at least one comma. If you see one or
more commas at the end of such lines, then your spreadsheet has been saved
correctly. You will not need to do this check again.
App 31162 stderr: However, if you’re seeing entries like these,
with empty final fields but no final comma(s), then you will need to download one of the free
Office suites for your FreeReg transcriptions. (Experienced volunteers who
use old versions of MS Excel and the "dummy column" method without any
problems may continue to do so, but we urge you to consider using a
different spreadsheet, if possible.) If you are more comfortable with database
software, then any that can save files in the required "Comma-separated
values" (CSV) format will be suitable. You will need to add the headers to
your completed CSV file, as described for spreadsheets. If your software
has AutoCorrect or Substitution options, you need to turn off all that
apply to punctuation. This method for FreeReg transcription is really
only for people who use a text editor for everything and wouldn't dream of
using anything else. You would have to set everything up yourself, be good
at counting commas and remember to quote fields that themselves contain
commas. (Full details can be found on <a
href="/cms/information-for-transcribers/entering-data-from-registers">Entering
data from registers.) You would also have to enter the file Headers
yourself, as described on <a
href="/cms/information-for-transcribers/file-headers">File headers. Plain text editors are no longer quite so plain.
In par
App 31162 stderr: ticular, they may have an AutoCorrect or Substitution
feature: you need to turn off all substitutions that apply to
punctuation, especially for quotation marks, ellipses and hyphens/dashes.
This applies even if you are just using a text editor to make a small edit
of your file or someone else's file. Please check that you know which character set
your editor uses on saving: if offered a choice, then select utf-8 and put
UTF-8 in line 1, field 6 (character set) of the file Headers.… ,BAIRD,,,,
[0m
App 31162 stderr: [32m+ … ,BAIRD,,,,
[0m
App 31162 stderr: … ,REYNOLDES,JONES,,Blacksmith,… ,BAIRD
[0m
App 31162 stderr: [32m+ … ,BAIRD
[0m
App 31162 stderr: … ,REYNOLDES,JONES,,Blacksmith
[0m
App 31162 stderr: [32m+
[0m
App 31162 stderr:
App 31162 stderr: Database
App 31162 stderr:
[0m
App 31162 stderr: [32m+
[0m
App 31162 stderr:
App 31162 stderr: Text editor
App 31162 stderr:
Kirk Dawson 5220 Riverside Drive Fairmont Hot Springs, B.C. V0B 1L1
On 3 January 2017 at 13:11, dougkdev notifications@github.com wrote:
What time was the passenger error?
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/FreeUKGen/MyopicVicar/issues/1049#issuecomment-270211701, or mute the thread https://github.com/notifications/unsubscribe-auth/ACvdRtb4aRJYuZNDuoUu_nwpwWfTtHz9ks5rOqtUgaJpZM4LZSzp .
Captainkirkdawson
commented
7 years ago 2 remaining issues Refinery assets still missing Some css still missing
richpomfret
commented
7 years ago To attempt to replicate.
Captainkirkdawson
commented
7 years ago I have found the cause. The public folder and all its contents had been moved into the lib folder, They have been moved back to where they should be and the system on colobus is now running as it was previously. Nothing more needs to be done. What was the cause. Clearly nothing to do with Refinery. Given the time it was most likely me. I use a gui to view the contents of the server. This enables me to download and upload files. Mostly of the logs and reports. I does have the ability for me to rename and drag and drop files or folders. Most of these such actions the server will not respond to because I do not have the correct permission and I cannot bypass the permission with a sudo. However I found in "retrieving the public folder" that the gui would move under different permission settings.
Bottom line it was my error. I need to be more careful with the gui
At 00:30 GMT on Jan 3, 2016 i did a restart.txt on coloubus to change a sleep parameter in mongo_config.yml. On accessing colobus I found that the css was missing. An Apache restart showed that the public directory was missing. Stopped server access for an hr while conducting investigation. Rebuilt the public assets. Did not include anything else. Server back in production. Further clean up will be required later in the say after some sleep