search

nolanlab / bead-normalization

bead-based normalization of mass cytometry data
GNU General Public License v3.0
13 stars 8 forks source link

random error while executing datenum #9

Closed nouteridgegit closed 8 years ago

nouteridgegit commented 8 years ago

Hello,

on behalf of a colleague of mine:

1) The problem: "I have a data acquired with a CyTOF 1 and instrument control software 6.0.626 and I would like to normalize them with MATlab Normalizer R2013b_MacOSX. Some of the files work when I try to normalize them separatly but when I try to normalize them all together I will get an error.

Some of them give right away an error. In both cases the error is the same as followed:

MATLAB:datenum:ConvertDateString DATENUM failed. datenumget_fcs_order111rderFilenames normalize_folder line 11

We tried to rename the files but doesn't help.

Does anybody know what the error means and can help? Would be awesome."

2) Environment: The machine the normalizer is executed on:

3) Guessing: It smells like being related to a Matlab classic as documented over here: http://www.mathworks.com/matlabcentral/newsreader/view_thread/287922#841230

... which then would turn this into a Matlab issue...

It looks like someone knowing the code of the normalizer could pinpoint the issue.

Would somebody please be so kind as to comment on this?

Best

rfinck commented 8 years ago

My guess is that there is an unexpected format of the timestamp in the header of the FCS files. Two possible ways to check:

library(flowCore)
fcs <- read.FCS(path/to/your/file)
description(fcs)['$BTIM']
description(fcs)['$DATE']
less path/to/your/file

Please copy and past the output from either of the above methods here.

nouteridgegit commented 8 years ago

Hello,

sorry for the delay and thank you for your reply!

I scanned 2 samples, one not causing issues, the other doing so.

This is what i cropped:

a) a file causing issue: `/////////////////////////////////////////// ar:~ account$ less /Volumes/Cytof/admin/Nolanlab-Normalizer_Datenum--knownbad_samples_160809/A1.fcs

FCS3.0 98 2989 299086432173 0 0 86432174 262936877/$CYT/DVSSCIENCES-CYTOF-6.0.626/$CYTSN/cytof2/$DATE/16-Mrz-2016/$FIL/S2.txt/$BTIM/14:04:18/$ETIM/14:33:33/$MODE/L/$DATATYPE/F/$BYTEORD/1,2,3,4/$PAR/48/$TOT/450152/$BEGINSTEXT/0/$ENDSTEXT/0/$NEXTDATA/0/$BEGINANALYSIS/0/$ENDANALYSIS/0/$BEGINDATA/2990/$ENDDATA/86432173/$P1N/Time/$P1B/32/$P1E/0,0/$P1R/2097152/$P2N/Event_length/$P2B/32/$P2E/0,0/$P2R/4096/$P3N/Xe131Di/$P3B/32/$P3E/0,0/$P3R/4096/$P4N/Cs133Di/$P4B/32/$P4E/0,0/$P4R/4096/$P5N/Ba138Di/$P5B/32/$P5E/0,0/$P5R/16384/$P6N/La139Di/$P6B/32/$P6E/0,0/$P6R/4096/$P7N/Ce140Di/$P7S/Beads_Ce140/$P7B/32/$P7E/0,0/$P7R/8192/$P8N/Pr141Di/$P8S/CD45_Pr141/$P8B/32/$P8E/0,0/$P8R/16384/$P9N/Nd142Di/$P9S/CD11a_Nd142/$P9B/32/$P9E/0,0/$P9R/8192/$P10N/Nd143Di/$P10S/HLADR_Nd143/$P10B/32/$P10E/0,0/$P10R/8192/$P11N/Nd144Di/$P11S/CD11b_Nd144/$P11B/32/$P11E/0,0/$P11R/16384/$P12N/Nd145Di/$P12S/C5L2_Nd145/$P12B/32/$P12E/0,0/$P12R/8192/$P13N/Nd146Di/$P13S/CD64_Nd146/$P13B/32/$P13E/0,0/$P13R/16384/$P14N/Sm147Di/$P14S/CD88_Sm147/$P14B/32/$P14E/0,0/$P14R/4096/$P15N/Nd148Di/$P15S/IL17A_148Nd/$P15B/32/$P15E/0,0/$P15R/4096/$P16N/Sm149Di/$P16S/CD66b_Sm149/$P16B/32/$P16E/0,0/$P16R/4096/$P17N/Nd150Di/$P17B/32/$P17E/0,0/$P17R/4096/$P18N/Eu151Di/$P18S/TNFa_Beads_Eu151/$P18B/32/$P18E/0,0/$P18R/4096/$P19N/Sm152Di/$P19S/AktP_152Sm/$P19B/32/$P19E/0,0/$P19R/8192/$P20N/Eu153Di/$P20S/Beads_Eu153/$P20B/32/$P20E/0,0/$P20R/4096/$P21N/Sm154Di/$P21B/32/$P21E/0,0/$P21R/4096/$P22N/Gd155Di/$P22B/32/$P22E/0,0/$P22R/4096/$P23N/Gd156Di/$P23S/p38P_Gd156/$P23B/32/$P23E/0,0/$P23R/4096/$P24N/Gd157Di/$P24B/32/$P24E/0,0/$P24R/8192/$P25N/Gd158Di/$P25S/IL1b_158Gd/$P25B/32/$P25E/0,0/$P25R/16384/$P26N/Tb159Di/$P26S/CD11_Tb159/$P26B/32/$P26E/0,0/$P26R/8192/$P27N/Gd160Di/$P27B/32/$P27E/0,0/$P27R/4096/$P28N/Dy161Di/$P28S/MHCII_Dy161/$P28B/32/$P28E/0,0/$P28R/16384/$P29N/Dy162Di/$P29S/CD32_Dy162/$P29B/32/$P29E/0,0/$P29R/8192/$P30N/Dy163Di/$P30S/CD18_Dy163/$P30B/32/$P30E/0,0/$P30R/4096/$P31N/Dy164Di/$P31S/CD15_Dy164/$P31B/32/$P31E/0,0/$P31R/8192/$P32N/Ho165Di/$P32S/CD16_Beads_Ho165/$P32B/32/$P32E/0,0/$P32R/8192/$P33N/Er166Di/$P33S/IL17F_166Er/$P33B/32/$P33E/0,0/$P33R/4096/$P34N/Er167Di/$P34S/CD35_Er167/$P34B/32/$P34E/0,0/$P34R/8192/$P35N/Er168Di/$P35B/32/$P35E/0,0/$P35R/8192/$P36N/Tm169Di/$P36S/CD33_Tm169/$P36B/32/$P36E/0,0/$P36R/8192/$P37N/Yb170Di/$P37B/32/$P37E/0,0/$P37R/8192/$P38N/Er170Di/$P38S/CD3_Er170/$P38B/32/$P38E/0,0/$P38R/8192/$P39N/Yb171Di/$P39S/CD20_Yb171/$P39B/32/$P39E/0,0/$P39R/4096/$P40N/Yb172Di/$P40B/32/$P40E/0,0/$P40R/4096/$P41N/Yb173Di/$P41S/CD177_Yb173/$P41B/32/$P41E/0,0/$P41R/8192/$P42N/Yb174Di/$P42B/32/$P42E/0,0/$P42R/4096/$P43N/Lu175Di/$P43S/CD14/$P43B/32/$P43E/0,0/$P43R/16384/$P44N/Yb176Di/$P44B/32/$P44E/0,0/$P44R/8192/$P45N/Os189Di/$P45

[ ... ] ///////////////////////////////////////////`

b) a file not causing issue: `/////////////////////////////////////////// Last login: Tue Aug 9 16:56:17 on ttys001

ar:~ account$ less /Volumes/Cytof/admin/Nolanlab-Normalizer_Datenum--knowngood_samples_160809/A3.fcs

FCS3.0 98 2999 300078890807 0 0 78890808 239998536/$CYT/DVSSCIENCES-CYTOF-6.0.626/$CYTSN/cytof2/$DATE/06-Apr-2016/$FIL/EF Exp20160406 S5a.txt/$BTIM/16:02:02/$ETIM/16:39:12/$MODE/L/$DATATYPE/F/$BYTEORD/1,2,3,4/$PAR/48/$TOT/410874/$BEGINSTEXT/0/$ENDSTEXT/0/$NEXTDATA/0/$BEGINANALYSIS/0/$ENDANALYSIS/0/$BEGINDATA/3000/$ENDDATA/78890807/$P1N/Time/$P1B/32/$P1E/0,0/$P1R/4194304/$P2N/Event_length/$P2B/32/$P2E/0,0/$P2R/4096/$P3N/Xe131Di/$P3B/32/$P3E/0,0/$P3R/4096/$P4N/Cs133Di/$P4B/32/$P4E/0,0/$P4R/4096/$P5N/Ba138Di/$P5B/32/$P5E/0,0/$P5R/4096/$P6N/La139Di/$P6B/32/$P6E/0,0/$P6R/8192/$P7N/Ce140Di/$P7S/Beads_Ce140/$P7B/32/$P7E/0,0/$P7R/8192/$P8N/Pr141Di/$P8S/CD45_Pr141/$P8B/32/$P8E/0,0/$P8R/16384/$P9N/Nd142Di/$P9S/CD11a_Nd142/$P9B/32/$P9E/0,0/$P9R/8192/$P10N/Nd143Di/$P10S/HLADR_Nd143/$P10B/32/$P10E/0,0/$P10R/4096/$P11N/Nd144Di/$P11S/CD11b_Nd144/$P11B/32/$P11E/0,0/$P11R/8192/$P12N/Nd145Di/$P12S/C5L2_Nd145/$P12B/32/$P12E/0,0/$P12R/8192/$P13N/Nd146Di/$P13S/CD64_Nd146/$P13B/32/$P13E/0,0/$P13R/8192/$P14N/Sm147Di/$P14S/CD88_Sm147/$P14B/32/$P14E/0,0/$P14R/8192/$P15N/Nd148Di/$P15S/IL17A_148Nd/$P15B/32/$P15E/0,0/$P15R/4096/$P16N/Sm149Di/$P16S/CD66b_Sm149/$P16B/32/$P16E/0,0/$P16R/4096/$P17N/Nd150Di/$P17B/32/$P17E/0,0/$P17R/4096/$P18N/Eu151Di/$P18S/TNFa_Beads_Eu151/$P18B/32/$P18E/0,0/$P18R/4096/$P19N/Sm152Di/$P19S/AktP_152Sm/$P19B/32/$P19E/0,0/$P19R/8192/$P20N/Eu153Di/$P20S/Beads_Eu153/$P20B/32/$P20E/0,0/$P20R/4096/$P21N/Sm154Di/$P21B/32/$P21E/0,0/$P21R/4096/$P22N/Gd155Di/$P22B/32/$P22E/0,0/$P22R/4096/$P23N/Gd156Di/$P23S/p38P_Gd156/$P23B/32/$P23E/0,0/$P23R/8192/$P24N/Gd157Di/$P24B/32/$P24E/0,0/$P24R/4096/$P25N/Gd158Di/$P25S/IL1b_158Gd/$P25B/32/$P25E/0,0/$P25R/8192/$P26N/Tb159Di/$P26S/CD11_Tb159/$P26B/32/$P26E/0,0/$P26R/4096/$P27N/Gd160Di/$P27B/32/$P27E/0,0/$P27R/4096/$P28N/Dy161Di/$P28S/MHCII_Dy161/$P28B/32/$P28E/0,0/$P28R/4096/$P29N/Dy162Di/$P29S/CD32_Dy162/$P29B/32/$P29E/0,0/$P29R/8192/$P30N/Dy163Di/$P30S/CD18_Dy163/$P30B/32/$P30E/0,0/$P30R/8192/$P31N/Dy164Di/$P31S/CD15_Dy164/$P31B/32/$P31E/0,0/$P31R/4096/$P32N/Ho165Di/$P32S/CD16_Beads_Ho165/$P32B/32/$P32E/0,0/$P32R/4096/$P33N/Er166Di/$P33S/IL17F_166Er/$P33B/32/$P33E/0,0/$P33R/4096/$P34N/Er167Di/$P34S/CD35_Er167/$P34B/32/$P34E/0,0/$P34R/8192/$P35N/Er168Di/$P35B/32/$P35E/0,0/$P35R/4096/$P36N/Tm169Di/$P36S/CD33_Tm169/$P36B/32/$P36E/0,0/$P36R/4096/$P37N/Yb170Di/$P37B/32/$P37E/0,0/$P37R/4096/$P38N/Er170Di/$P38S/CD3_Er170/$P38B/32/$P38E/0,0/$P38R/4096/$P39N/Yb171Di/$P39S/CD20_Yb171/$P39B/32/$P39E/0,0/$P39R/4096/$P40N/Yb172Di/$P40B/32/$P40E/0,0/$P40R/4096/$P41N/Yb173Di/$P41S/CD177_Yb173/$P41B/32/$P41E/0,0/$P41R/16384/$P42N/Yb174Di/$P42B/32/$P42E/0,0/$P42R/8192/$P43N/Lu175Di/$P43S/CD14/$P43B/32/$P43E/0,0/$P43R/8192/$P44N/Yb176Di/$P44B/32/$P44E/0,0/$P44R/8192/$P45N/Os189Di/$P45B/32/$P45E/0,0/$P45R/4096/$P46N/Ir191Di/$P46S/DNA_Ir191/$P46B/32/$P46E/0,0/$P46R/4096/$P47N/Ir193Di/$P47S/DNA_Ir193/$P47B/32/$P47E/0,0/$P47R/8192/$P48N/Pt195Di/$P48S/Live_Dead/$P48B/32/$P48E/0,0/$P48R/8192

[ ... ] ///////////////////////////////////////////`

Please forgive the plenty of output, i just wanted to make sure...

In terms of $BTIM the current locale may matter, since it adheres to the european time format; on the other hand, thats true for all the files...

Do the samples provide any clue?

Best

rfinck commented 8 years ago

Yes, I think the locale is the issue, and that the files whose month abbreviation matches the English (e.g. Apr) are the ones that work, whereas the files whose month abbreviations are different do not (e.g. Mrz/Mar).

This is a bug. You should be able to use a non-English language.

Are you using the normalizer in MATLAB directly or one of the compiled applications?

On Aug 9, 2016 8:47 AM, "nouteridgegit" notifications@github.com wrote:

Hello,

sorry for the delay and thank you for your reply!

I scanned 2 samples, one not causing issues, the other doing so.

This is what i cropped:

a) a file causing issue: `/////////////////////////////////////////// ar:~ account$ less /Volumes/Cytof/admin/Nolanlab- Normalizer_Datenum--knownbad_samples_160809/A1.fcs

FCS3.0 98 2989 299086432173 0 0 86432174 262936877/$CYT/DVSSCIENCES- CYTOF-6.0.626/$CYTSN/cytof2/$DATE/16-Mrz-2016/$FIL/S2.txt/$ BTIM/14:04:18/$ETIM/14:33:33/$MODE/L/$DATATYPE/F/$BYTEORD/1, 2,3,4/$PAR/48/$TOT/450152/$BEGINSTEXT/0/$ENDSTEXT/0/$ NEXTDATA/0/$BEGINANALYSIS/0/$ENDANALYSIS/0/$BEGINDATA/2990/ $ENDDATA/86432173/$P1N/Time/$P1B/32/$P1E/0,0/$P1R/2097152/$ P2N/Eventlength/$P2B/32/$P2E/0,0/$P2R/4096/$P3N/Xe131Di/$ P3B/32/$P3E/0,0/$P3R/4096/$P4N/Cs133Di/$P4B/32/$P4E/0,0/$ P4R/4096/$P5N/Ba138Di/$P5B/32/$P5E/0,0/$P5R/16384/$P6N/ La139Di/$P6B/32/$P6E/0,0/$P6R/4096/$P7N/Ce140Di/$P7S/Beads Ce140/$P7B/32/$P7E/0,0/$P7R/8192/$P8N/Pr141Di/$P8S/CD45 Pr141/$P8B/32/$P8E/0,0/$P8R/16384/$P9N/Nd142Di/$P9S/CD11a Nd142/$P9B/32/$P9E/0,0/$P9R/8192/$P10N/Nd143Di/$P10S/ HLADR_Nd143/$P10B/32/$P10E/0,0/$P10R/8192/$P11N/Nd144Di/$ P11S/CD11b_Nd144/$P11B/32/$P11E/0,0/$P11R/16384/$P12N/ Nd145Di/$P12S/C5L2_Nd145/$P12B/32/$P12E/0,0/$P12R/8192/$ P13N/Nd146Di/$P13S/CD64_Nd146/$P13B/32/$P13E/0,0/$P13R/ 16384/$P14N/Sm147Di/$P14S/CD88_Sm147/$P14B/32/$P14E/0,0/ $P14R/4096/$P15N/Nd148Di/$P15S/IL17A_148Nd/$P15B/32/$ P15E/0,0/$P15R/4096/$P16N/Sm149Di/$P16S/CD66b_Sm149/$ P16B/32/$P16E/0,0/$P16R/4096/$P17N/Nd150Di/$P17B/32/$P17E/0, 0/$P17R/4096/$P18N/Eu151Di/$P18S/TNFa_Beads_Eu151/$P18B/ 32/$P18E/0,0/$P18R/4096/$P19N/Sm152Di/$P19S/AktP152Sm/$ P19B/32/$P19E/0,0/$P19R/8192/$P20N/Eu153Di/$P20S/Beads Eu153/$P20B/32/$P20E/0,0/$P20R/4096/$P21N/Sm154Di/$P21B/ 32/$P21E/0,0/$P21R/4096/$P22N/Gd155Di/$P22B/32/$P22E/0,0/$ P22R/4096/$P23N/Gd156Di/$P23S/p38P_Gd156/$P23B/32/$P23E/0,0/ $P23R/4096/$P24N/Gd157Di/$P24B/32/$P24E/0,0/$P24R/8192/$ P25N/Gd158Di/$P25S/IL1b_158Gd/$P25B/32/$P25E/0,0/$P25R/ 16384/$P26N/Tb159Di/$P26S/CD11_Tb159/$P26B/32/$P26E/0,0/ $P26R/8192/$P27N/Gd160Di/$P27B/32/$P27E/0,0/$P27R/4096/$ P28N/Dy161Di/$P28S/MHCII_Dy161/$P28B/32/$P28E/0,0/$ P28R/16384/$P29N/Dy162Di/$P29S/CD32_Dy162/$P29B/32/$ P29E/0,0/$P29R/8192/$P30N/Dy163Di/$P30S/CD18_Dy163/$ P30B/32/$P30E/0,0/$P30R/4096/$P31N/Dy164Di/$P31S/CD15Dy164/ $P31B/32/$P31E/0,0/$P31R/8192/$P32N/Ho165Di/$P32S/CD16 Beads_Ho165/$P32B/32/$P32E/0,0/$P32R/8192/$P33N/Er166Di/$ P33S/IL17F_166Er/$P33B/32/$P33E/0,0/$P33R/4096/$P34N/ Er167Di/$P34S/CD35_Er167/$P34B/32/$P34E/0,0/$P34R/8192/$ P35N/Er168Di/$P35B/32/$P35E/0,0/$P35R/8192/$P36N/Tm169Di/$ P36S/CD33_Tm169/$P36B/32/$P36E/0,0/$P36R/8192/$P37N/ Yb170Di/$P37B/32/$P37E/0,0/$P37R/8192/$P38N/Er170Di/$P38S/ CD3_Er170/$P38B/32/$P38E/0,0/$P38R/8192/$P39N/Yb171Di/$P39S/ CD20Yb171/$P39B/32/$P39E/0,0/$P39R/4096/$P40N/Yb172Di/$ P40B/32/$P40E/0,0/$P40R/4096/$P41N/Yb173Di/$P41S/CD177 Yb173/$P41B/32/$P41E/0,0/$P41R/8192/$P42N/Yb174Di/$P42B/ 32/$P42E/0,0/$P42R/4096/$P43N/Lu175Di/$P43S/CD14/$P43B/32/$ P43E/0,0/$P43R/16384/$P44N/Yb176Di/$P44B/32/$P44E/0,0/$ P44R/8192/$P45N/Os189Di/$P45

[ ... ] ///////////////////////////////////////////`

b) a file not causing issue: `/////////////////////////////////////////// Last login: Tue Aug 9 16:56:17 on ttys001

ar:~ account$ less /Volumes/Cytof/admin/Nolanlab- Normalizer_Datenum--knowngood_samples_160809/A3.fcs

FCS3.0 98 2999 300078890807 0 0 78890808 239998536/$CYT/DVSSCIENCES- CYTOF-6.0.626/$CYTSN/cytof2/$DATE/06-Apr-2016/$FIL/EF Exp20160406 S5a.txt/$BTIM/16:02:02/$ETIM/16:39:12/$MODE/L/$DATATYPE/F/$ BYTEORD/1,2,3,4/$PAR/48/$TOT/410874/$BEGINSTEXT/0/$ ENDSTEXT/0/$NEXTDATA/0/$BEGINANALYSIS/0/$ENDANALYSIS/ 0/$BEGINDATA/3000/$ENDDATA/78890807/$P1N/Time/$P1B/32/$ P1E/0,0/$P1R/4194304/$P2N/Event_length/$P2B/32/$P2E/0,0/ $P2R/4096/$P3N/Xe131Di/$P3B/32/$P3E/0,0/$P3R/4096/$P4N/ Cs133Di/$P4B/32/$P4E/0,0/$P4R/4096/$P5N/Ba138Di/$P5B/32/$ P5E/0,0/$P5R/4096/$P6N/La139Di/$P6B/32/$P6E/0,0/$P6R/ 8192/$P7N/Ce140Di/$P7S/Beads_Ce140/$P7B/32/$P7E/0,0/$P7R/ 8192/$P8N/Pr141Di/$P8S/CD45_Pr141/$P8B/32/$P8E/0,0/$P8R/ 16384/$P9N/Nd142Di/$P9S/CD11a_Nd142/$P9B/32/$P9E/0,0/$P9R/ 8192/$P10N/Nd143Di/$P10S/HLADR_Nd143/$P10B/32/$P10E/0, 0/$P10R/4096/$P11N/Nd144Di/$P11S/CD11b_Nd144/$P11B/32/$ P11E/0,0/$P11R/8192/$P12N/Nd145Di/$P12S/C5L2_Nd145/$ P12B/32/$P12E/0,0/$P12R/8192/$P13N/Nd146Di/$P13S/CD64Nd146/ $P13B/32/$P13E/0,0/$P13R/8192/$P14N/Sm147Di/$P14S/CD88 Sm147/$P14B/32/$P14E/0,0/$P14R/8192/$P15N/Nd148Di/$P15S/ IL17A_148Nd/$P15B/32/$P15E/0,0/$P15R/4096/$P16N/Sm149Di/$ P16S/CD66b_Sm149/$P16B/32/$P16E/0,0/$P16R/4096/$P17N/ Nd150Di/$P17B/32/$P17E/0,0/$P17R/4096/$P18N/Eu151Di/$P18S/ TNFa_Beads_Eu151/$P18B/32/$P18E/0,0/$P18R/4096/$P19N/ Sm152Di/$P19S/AktP_152Sm/$P19B/32/$P19E/0,0/$P19R/8192/$ P20N/Eu153Di/$P20S/Beads_Eu153/$P20B/32/$P20E/0,0/$ P20R/4096/$P21N/Sm154Di/$P21B/32/$P21E/0,0/$P21R/4096/$P22N/ Gd155Di/$P22B/32/$P22E/0,0/$P22R/4096/$P23N/Gd156Di/$P23S/ p38P_Gd156/$P23B/32/$P23E/0,0/$P23R/8192/$P24N/Gd157Di/$ P24B/32/$P24E/0,0/$P24R/4096/$P25N/Gd158Di/$P25S/IL1b158Gd/ $P25B/32/$P25E/0,0/$P25R/8192/$P26N/Tb159Di/$P26S/CD11 Tb159/$P26B/32/$P26E/0,0/$P26R/4096/$P27N/Gd160Di/$P27B/ 32/$P27E/0,0/$P27R/4096/$P28N/Dy161Di/$P28S/MHCII_Dy161/$ P28B/32/$P28E/0,0/$P28R/4096/$P29N/Dy162Di/$P29S/CD32Dy162/ $P29B/32/$P29E/0,0/$P29R/8192/$P30N/Dy163Di/$P30S/CD18 Dy163/$P30B/32/$P30E/0,0/$P30R/8192/$P31N/Dy164Di/$P31S/ CD15_Dy164/$P31B/32/$P31E/0,0/$P31R/4096/$P32N/Ho165Di/$ P32S/CD16_Beads_Ho165/$P32B/32/$P32E/0,0/$P32R/4096/$P33N/ Er166Di/$P33S/IL17F_166Er/$P33B/32/$P33E/0,0/$P33R/4096/$ P34N/Er167Di/$P34S/CD35_Er167/$P34B/32/$P34E/0,0/$P34R/8192/ $P35N/Er168Di/$P35B/32/$P35E/0,0/$P35R/4096/$P36N/Tm169Di/$ P36S/CD33_Tm169/$P36B/32/$P36E/0,0/$P36R/4096/$P37N/ Yb170Di/$P37B/32/$P37E/0,0/$P37R/4096/$P38N/Er170Di/$P38S/ CD3_Er170/$P38B/32/$P38E/0,0/$P38R/4096/$P39N/Yb171Di/$P39S/ CD20Yb171/$P39B/32/$P39E/0,0/$P39R/4096/$P40N/Yb172Di/$ P40B/32/$P40E/0,0/$P40R/4096/$P41N/Yb173Di/$P41S/CD177 Yb173/$P41B/32/$P41E/0,0/$P41R/16384/$P42N/Yb174Di/$ P42B/32/$P42E/0,0/$P42R/8192/$P43N/Lu175Di/$P43S/CD14/$P43B/ 32/$P43E/0,0/$P43R/8192/$P44N/Yb176Di/$P44B/32/$P44E/0,0/$ P44R/8192/$P45N/Os189Di/$P45B/32/$P45E/0,0/$P45R/4096/$P46N/ Ir191Di/$P46S/DNA_Ir191/$P46B/32/$P46E/0,0/$P46R/4096/$P47N/ Ir193Di/$P47S/DNA_Ir193/$P47B/32/$P47E/0,0/$P47R/8192/$P48N/ Pt195Di/$P48S/Live_Dead/$P48B/32/$P48E/0,0/$P48R/8192

[ ... ] ///////////////////////////////////////////`

Please forgive the plenty of output, i just wanted to make sure...

In terms of $BTIM the current locale may matter, since it adheres to the european time format; on the other hand, thats true for all the files...

Do the samples provide any clue?

Best

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nolanlab/bead-normalization/issues/9#issuecomment-238596487, or mute the thread https://github.com/notifications/unsubscribe-auth/ABImwN6YOElafO-aU-4JyqtVgEikkcwaks5qeKEPgaJpZM4Jd3Nz .

nouteridgegit commented 8 years ago

Hello R,

brilliant, and thanks again for your quick reply.

We use the binary provided by NormalizerR2013b_MacOSX.zip, thus i think we use the complied application?

The actual trigger may be the current config of the machine the analyzer is run on:

`/////////////////////////////////////////// Last login: Tue Aug 9 22:44:32 on console ar:~ account $ locale LANG= LC_COLLATE="C" LC_CTYPE="UTF-8" LC_MESSAGES="C" LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_ALL= ar:~ account $

Region: Germany Calendar: Gregorian Format Language: English ///////////////////////////////////////////`

I expect switching to "Format Language: German" has potential to fix it.

As mentioned: I'm investigating this system problem on behalf of my research colleague, who is off for holidays as of today, thus verification may not be immediate.

If i manage to reproduce my first colleague's procedure for normalization with another research colleague, i'll post the result soon. Otherwise: a bit later ;-)

Best

nouteridgegit commented 8 years ago

Hello,

i just wanted to feed back that pinning all environment variables concerning the date format to german conventions and running the same original normalizing procedure using the compiled application ("NormalizerR2013b_MacOSX.app") on the same files and same machine regrettable did not help the issue.

Are there any options (CLI switches) available to verify that the application is sucessfully picking up the [changed] locale from the OS?

Best

rfinck commented 8 years ago

Here are Matlab's instructions on how to set the locale for OSX, Windows, and Linux: http://www.mathworks.com/help/matlab/internationalization.html

I just did a quick test where I reproduced your error on an FCS file with a date whose month was 'Mai', and then fixed the error by following the above instructions for OSX and changing my locale/language to German. I have not tested other platforms.

You can also, if you get desperate, rewrite the month of your FCS files, but hopefully that is not necessary.

On Thu, Aug 18, 2016 at 6:57 AM, nouteridgegit notifications@github.com wrote:

Hello,

i just wanted to feed back that pinning all environment variables concerning the date format to german conventions and running the same original normalizing procedure using the compiled application ("NormalizerR2013b_MacOSX.app") on the same files and same machine regrettable did not help the issue.

Are there any options (CLI switches) available to verify that the application is sucessfully picking up the [changed] locale from the OS?

Best

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/nolanlab/bead-normalization/issues/9#issuecomment-240730867, or mute the thread https://github.com/notifications/unsubscribe-auth/ABImwFq9niRDp_YQkp8xsOAVzRzx28jjks5qhGTdgaJpZM4Jd3Nz .

nouteridgegit commented 8 years ago

Hello R,

thank you very much for keeping track!

Thats interesting: i already had applied those measures. And doublechecked now, having read your last answer, switching back and forth between US_en and DE_de. It still doesn't help the issue or me. Possibly OS X 10.9.5 (what i used for testing) behaves differently here, i'll check that also.

Since i wouldn't like to tinker with the local setting on a global scale (managed machines), i prefer to fix the date string using an hex editor anyways - and that helped definitely.

I.e.: the issue is verified.

And that justifies to fix it at it's root: we'll investigate how to disable the acquisition application's behaviour to write a localized date string into the headers.

And we'll find means to fix the string in files already acquired using a script based approach.

Thanks again for your help,

Best

rfinck commented 8 years ago

It is unfortunate that the CyTOF software's convention is not to write datetimes in ISO 8601. Of course, if they were to make that switch now, one would also have to update the normalization software because it does currently make an assumption using the current the datetime format.

If you need to update the headers of pre-existing files, one way is to use the R package cytofCore. That's how I faked a file with the month of Mai.

library(cytofCore)
fcs <- read.FCS('path/to/fcs/file.fcs')
fcs_channels <- parameters(fcs)$desc
fcs_description <- description(fcs)
fcs_description['$DATE'] <- date_string_in_correct_format
cytofCore.write.FCS(exprs(x), 'path/to/new/fcs/file.fcs', channelDescriptions=fcs_channels,
                    referenceDescription=fcs_description, oldDescription=fcs_description)

Actually converting the German dates to the English dates is left as an exercise. ;)

One possibility for why I was able to process German dates whereas you were not is that when I run NormalizerR2013b_MacOSX.app, it might be calling an MCR that came with a full MATLAB installation. I could make sure it's calling the standalone MCR and possibly reproduce your continued error, but that does not provide a fix. Long-term, I'd like to port the normalizer to python to make it easier to maintain and address such issues as these.

nouteridgegit commented 8 years ago

Hello,

please forgive my delay.

Thank you for putting things into perspective, and for your proposal for updating the header files.

We'll check out this method once our researcher is back.

@MCR: that makes a lot of sense.

But honestly, this is exactly why i am keen on having our deployment to use an english UI and ENV whereever possible; the CyTOF's acquisition machine is beyond my scope ;-)

I'd like to also offer our users to patch the files already recorded in a programmatic fashion, in case they don't want to bother with it. This is what i wrote up as a preflight and it seems to do the job:

<code>
for i in `find . -type f -name '*.fcs'`;do \
hexdump -ve '1/1 "%.2X"' $i| \
sed "s/2D4D727A2D/2D4D61722D/g; \
s/2D4D61692D/2D4D61692D/g; \
s/2D4F6B742D/2D4F63742D/g; \
s/2D44657A2D/2D4465632D/g"| \
xxd -r -p > $i.patched ; done
</code>

Don't look too sharp at the for/find construction, i know, i've been lazy ;-)

It's inherently inefficent, as it has to write out the entire stream, so quitting after substituting the first (relevant) instance is not an option apparently...

Best

nouteridgegit commented 8 years ago

Hello R,

sorry, just got aware that this issue is still listed open. From our perspective it actually could be closed? Both the aforementioned fixes do their job, thus FCS files then can sucessfully be nomalized using the:

Thank you very much again for your help!

Best

rfinck commented 8 years ago

Great, thanks for keeping track of this.