Closed stevemarin closed 4 years ago
ellarabi
commented
4 years ago Hi! Thank you for raising this issue.
I'd assume that has to do with (in)accuracy of the Polyglot model we made use of. The monolingual part of the dataset should ideally be in English (without additional languages being mixed in). Do you have a rough estimate of what ratio of the monolingual data is actually code-switched? We would consider cleaning up that data, in case it's "contaminated" significantly.
stevemarin
commented
4 years ago So I noticed the issue by visually inspecting some of the monolingual data just using a cat | shuf | head pipeline in the terminal and a lot of CS-looking entries were popping up. To determine a rough estimate, I randomly selected 100 entries and checked which were and were not monolingual. The results are below. I tried to be fair to the rules in your paper. I made an effort to ignore proper noun and so forth. I still get about 34/100 appearing multilingual. My results are below:
0 | I know what he did is douchey but no one should be outed like that.
0 | No prob. It's more on software restriction yan ng PS4 than hardware. This is what I hated with Sony when they released PS4.
0 | How many VULs did you invest in exactly? And when did you start investing?
1 | Imma start treating bules like shit now to balance out the cornballs who treat them nice for no good reason.
0 | Yeah, hopefully not dying in some painful old-age disease. It would be nice to just close your eyes and go in an enteral slumber.
0 | Yes I pay exactly 1942 pesos for 5mbps + landline and 2099 pesos for 6mbps + landline.
1 | Read it as 'dyoga' also, fuck you puro ka Davao putangina ka pagandahin mo pilipinas
1 | Marami nmn choices u can go for tracks(that is if he's still using his tamiya) or chassis, engines or kung collector sya u could buy him a new one..although di ko lng sure kung may nagbebenta until now
0 | Fun Fact: fast food places in PH are considering including Chinese characters into their menus
0 | he doesn't own /u/the_yaya, only /u/phrdbot reddit was down for a little bit when /u/the_yaya was supposed to post
0 | You're just arguing about semantics now. It isn't like I was saying people are like Meursault in The Stranger.
0 | Or maybe Olive Oyl is just her alias, her real name is Rapeseed. 😲😲😲😲😲
1 | hahahah, that serta merta part. nice point you got there
1 | So another comment na lumiit din daw sa Greenhills and Rockwell. I'm so sad to hear this trend. :(
1 | Actually yes pero kasi sobrang small item lang niya and cheap kaya dito ko na directly pinaship
0 | Is it weird that I feel nostalgia after seeing this?
0 | How can anyone think that twice should only be for girls? I like wrestling but I love twice hahahaha
1 | Medu, arc warden, gyro, at sniper. NAPAKAHIRAP mag highground against them jusko hahaha
0 | Mainly because she handled the crowd quite well in the Jakarta governor debate...
0 | Same, 1 came in from one of the stops by the bus that are going to Cavite via Cavitex / Baclaran route, another one came in baclaran area. Both sell the same item and have different story but I was doubtful on the second dude since he looked like buffed and was more white / light skinned compared to the other guy that looked like skin and bones but dressed properly. But I bought pens since I need them lol. Next though, for anyone going to this said route,just ignore / don't give in, if you need pen just buy 1.
1 | They will bann u if u detected using "3rd-party-app" So maybe Grab worried if u custom their app. Padahal kalo edit apk di emulator juga bisa.
0 | well i think that's the ending. i mean she already have a la senza bag on her
1 | I made fun of our kasambahay’s spelling before. Coffee mate>coffee meat really laughed tf out but I hurt her feelings. She stopped putting labels on stuff in our kitchen 😞 Skl.
0 | neither, they not really want back to communism or living as one big country, they just disappoint that country which defeat fascism, was first to send machine, animal and then human to space break apart due incompetent leadership
0 | That's why I was surprised when I got here in Canada, In fast food restaurants like mcdonalds, everyone throws away their garbage, return back the tray, and even wipe their table with the paper towel if they need to do so. Also, when I was with my white friend and we can't find any trash bin at walmart, he asked one of the employees the location of garbage bins and the employee just told him to just hand it in to her as she will just throw it by herself at the back of her station.
1 | As a former OFW, i totally get this. pero ang problema din kasi sa madaming ofw they show off a lot on social media, making it appear that they're living the life (even if they're suffering behind the scenes). Tsaka madami talagang mayabang even face-to-face. So bawas bawas din sana.
1 | Make sure to pick an interesting movie, and not a boring one then. 🙂 Now I just thought of this... If you're really interested sa girl mag movie kayo para more topics to talk about, more time to spend together. May bwelo before mag dinner, and longer time spent together mas at ease na siguro kayo by the end of the date. Pag coffee date lang, tapos naging awkward na mas mahirap i-redeem ung self.
0 | Anyone know a printing shop that makes clear stickers? Thanks!
0 | Why does it look like Bagiuo got hit wayyyy harder than other areas?
1 | Not sure? Pero hotels with TV screens while resting Sa bath tub, meron. Hehe
1 | Well, I think that is because of abuse of the law. I mean DSWD does not have the budget to do this and on the other side of the law it's basically impossible to charge them. Not to mention the Philippine justice system is punitive. If they insist, then every nuisance person, child or adult, should just be killed na lang at sight. Mas patas kung ganun. /s
0 | Definitely true and they get annoyed when you don't let them sneak in as they are "in a hurry" and just give them a chance.
1 | Oo nga eh. Buti may bidet to clean, para di ma shookt yung next na makikiihi.
0 | Cold weather, tea and a nice playlist will never fail to make my day
0 | I hate to be that woman (is there even such a phrase? I digress) but I have to talk the right fact. The copy of Windows 10 you download from Microsoft will always be genuine. Are you a pirate? No. Are you running unlicensed copy? Yes. Are you wrong? Yes. Microsoft just can't be bothered with individual, so that's why. Personally, I can't be bothered with other's morale, but I care about mine. Therefore I purchased a legitimate copy of Windows 7, 8.1, 10 and Office 2013. It's a one-time purchase and I can use the license forever anyway.
1 | Am facut plat pe el.E genial. Crimele is un pic repetitive dar in rest ii super.
0 | stop generalizing and being corrupt is irrelevant because as i've said earlier, good immig officers of other good countries have denied entry for much less.
0 | The ones in sachet are available in some Mercury Drug stores. I got mine from their BGC branch.
0 | I don’t think I’ve heard Hiwaga before. Gonna check it out.
0 | So, what should we set the countdown timer for the "He was making a joke." statement?
0 | As someone in Manila, Let the shitstorm begin.... (now waiting for Villar to enter.)
0 | Now we're talking. But the stumbling block is this: a sizeable chunk of the population genuinely believe that this administration is good for them.
1 | I’ve read somewhere na majority ng mga basura sa Manila Bay galing sa coastline ng Cavite.
0 | For me, if the PS5 is backwards compatible with PS4 games and has God of War 2018 sequel or Horizon 2 as launch titles, I will buy it day one :))
0 | Avoid jeepney and fx, avoid LRT 1 and mrt since thru are crowded. Avoid bad areas, stay only in bgc or greenbelt area of Makati. And then you have some relief
0 | No worries I'm Malaysian, I'll probably blend in with them. I understand Bahasa Indonesia and I can speak Malaysian bahasa of course, and I speak Chinese too. So I don't think this will be an issue
0 | This millennium only started in 2001. Intolerance is on the rise everywhere in the world and has been since like 2014.
1 | Paano gawing "mas vibrant" yung wallpaper sa cellphone na may 401 ppi? The wallpaper is a 4k image na inedit sa photoshop with 300 dpi pero hindi pa rin siya nag pop unloke sa stock wallpaper ng phone.
1 | I wish may option sa Spotify na panay lyrics lang idisplay nila. *sigh*
0 | Or you can get unemployment check. Good for at least couple of months if you cant really find a job
0 | >penis executed Sigh... Fine... Put your penis inside the guillotine.
0 | Does anyone know any good SMALL phones (Less than 5.5 inches)? I know phones have one handed mode but I'd rather have the phone itself fit in my hand.
1 | Baka naman kasi bisexual (lang?). It's not uncommon for us gym fit males na malibugan (din?) sa kapwa gym fit. :P
1 | True. Learn to read data. Data science is the future. Also, malaking factor ang marunong sa excel, kahit vba scripting.
0 | Keen to see Davao activity for 2018. Fares going in and out of Davao is crazy expensive now!
0 | Birth control is free in America. Cant get any cheaper than that. And female brain here (with an copper IUD).. I DEFINITELY think I'll be having sex for as long as I can. Hence why I got it and most of my friends do too. Get out out of here with that notion that wanting to have sex in the long term is slutty. Total wtf.
0 | Yep. My doctor gave me a muscle relaxant before but I really don't want to rely on meds so I usually sleep it off, apply white flower/katinko/Vicks/whatever on the part it throbs the most.
0 | I don't always walk, but when I do I ride the bus part of the way and then walk ~2km (the distance from Estrella on EDSA to my office in BGC).
0 | Do you play online games with Sky? 3 years ago I heard it was shitty to play League with. I guess that just goes with PH servers.
1 | Ay okay thank you. Di ako aware na may ganyan palang subreddit. THanks.
0 | This. Smoking is a better escape mechanism than just swiping around my phone. It also would make it less awkward when you're dead end while hanging with friends.
1 | I understand. Im just thinking of the worst case scenario for OP. He wants to do something sa rapist, if he will be traced and asked about the reason why need nia ipabugbog - and end up saying something about the rape issue, if the victim decides to stay quiet - siya yung mababalikan,
1 | He's not even trying to be visible. Mas masipag pa magcomment sa relevant issues si Diokno eh pareho naman silang lawyer. He really should step up his online presence.
1 | Wow. Read your original post. Was about gov’t providing jobs for the people so they don’t have to go abroad right? Then next comment was about meaningful jobs and tambays. Now about people getting 4ps. Those are in the rural areas. They are not even on the job market. Your logic is really screwed up as you move the goalposts. Lol. Then label me as a propagandist when you have nothing else?
1 | damn, good to see people here on Reddit that has a common thinking about this society of ours! Joining Reddit was also my best decision so far when it comes presence in the cyber space, specially joining r/Philippines sub!!!!! Mabuhay ang mga totoong Pilipino na hindi lang sa Free Facebook umaasa! hahahahaha ​
0 | Not a new capital, just a city to move the richer folks out of Manila to try to decongest Metro Manila, after that, an anti poor version of Order 66 will happen
1 | > Benarkah Burung Prabowo sudah dipotong/dikebiri habis ? quality question right here
0 | I want to see the price list of this branch of Jollibee.
1 | I've been wanting to do that too ever since nag work na ako. Kaso nanghihinayang ako sa food na naiwan after the noche buena and new year
0 | Is it still freedom of expression if you peddle fake news?
1 | They do, lalo na kapag puno ng water lily ang Pasig River.
0 | KAPA - May sound like a scam at first but it's actually not as the process is visible. Too good to be true but results don't lie though. Local politicians and government employees, even the police and military have involvements there.
1 | Hi! Congrats! Expect trainings sa audit firm and yung sa work, ready ka lang sa mga ipapagawa nila, make sure to focus on time management :D
0 | Hahaha. > You brought SJW in the first place. Yes, my hatred is irrelevant. But not so for people who persecuted, harassed, doxed by these SJW rabbits. Yes, I brought it up, but I was talking about people like Maimon. Also remind me again who gets persecuted by SJWs? While I agree that nobody should be harassed, but I do believe people like Maimon should be doxxed. Now, back to Neely. > Nothing wrong speaking with your compatriot in your own language. I never said it did. Wrong isn't the word I used. But if there's other person who doesn't speak your language in your group, in my opinion, it's impolite and rather tribal. Neely wasn’t in the group. Neither were the unnamed staff members. They had nothing to do with the conversation. For example, my workmates are as diversed as it could be. I can speak in English with the expats while at the same time, just two meters away from us, another pair of people are speaking in Sundanese. You know why it’s not a problem, even when it’s in the office? Because one conversation has nothing to do with the other. And no one feels insulted by not being included in every conversation. Now that’s common decency. > Of course, there's no hard set of rules, but it's common decency. Example, if you and two of your friends (who are Chinese/any nationality) hang out together, and your friends speak in their own language all the time, I bet you will feel left out. If not, then, you're a better person than most people. Yeah, but again, Neely and the other unnamed staff members **WERE NOT** hanging out together with the students. Why do you keep ignoring this fact to suit your narrative? Why did they even bother in the first place? > For proficiency, there is no suggestion, but it's implied. It's also not uncommon that ones not fluent in language that is not their mother tongue. If you or these students think their English is perfect, then well... ignore the suggestion. Well if perfection in English was the necessary requirement, I would bet my right hand that more international students would fare much better than white American students. The international students actually study the language and have to pass the proficiency level. Most of them might not speak as comfortable as white Americans, but they do write better than most white Americans. > If there is official language for the said company, then yes, sending email would be appropriate if they're using french all the time in the office, which most likely what these Chinese students do. Even in some offices, they make a specific day for a certain language, like English Day, Indonesian Day, etc. for the sake of learning that language. Is there any official language for the university? While the medium of instruction is Indeed English, the students aren’t required to speak it all the time. Just like when you apply for a job at a multinational company, you are required to have proficiency in English. But does that keep you from getting a promotion when you speak Indonesian when there are non-Indonesian speakers in the vicinity? > From my experience, the Chinese students indeed do speak Chinese all the time and loudly. Generally, their English aren't that fluent either. And here is the problem, by pointing this out, I would be branded with every bad names in SJW's moral book. But the reality I saw, fact according my limited experience, which doesn't align with SJW doctrine. In real life, of course, I wouldn't say anything, but Dr. Neely is an educator. It's not uncommon for professor to give counsel to students, and that's exactly how I see these emails. University is place for learning, to improve oneself. Or, is it considered bad to receive criticize and better oneself? Well, by SJW's logic, it is. Others are wrong and SJW is always right. No middle ground. No compromise. The world should bend to their wishes and their alternate reality. Really, the speak Chinese loudly all the time as in class as well? Where do/did you go to school? I might be able to check for confirmation. Also fun fact, fluency in English while encouraged, is not an obligation. Just like any expat in Jakarta is most likely isn’t fluent in Indonesian either. They still get paid, and they still hold better positions than Indonesian staff. The fluency is not an obligation. > The consequences: if your communication/English skill isn't good enough then you won't get the position. That I call fair. I agree, but good communication skill is not equivalent to English fluency. Maybe you should try to get a job at multinational company. Then you’ll see that many expats that aren’t from the US or UK aren’t that fluent in English either, even when they come from European countries. Has it ever been a problem for them in getting a good position? No. > And English is the official language in some states in the US. But what matter is what language used in class. You can call it official or not, but if it's used as common language then in that particular professional settings, it is a official language. Again, the keywords, as you constantly put it, are “the professional settings. Which the students were not in. How hard is it for you to understand that their free time with their friends, outside of class session, is not a professional setting? Unless you understand that, there really is no point in arguing..
0 | From what I know you can develop a resistance to it if you get used to it. The hard part is that you have to regularly drink it (but I guess in small doses)
0 | I hired a freelance, but i forget this page...(As you can read in my post).
1 | I think reddit philippines should have a meetup for those in the US or west coast like me. Mukang marami naman.
0 | little detail for you : ​ ​ ​ mass = volume \* density ​ 1 m^(3) \* 997 kg/m³ = 997 kg ​ force = mass \* acceleration ​ 997 kg \* 10 m/s^(2) (assumption) = 9970 N ​ 1 kg = 9.81 N ​ : 1016.30 kg
0 | The McDo in C. Raymundo, Pasig also has this, but it's on the counter beside the cashiers.
0 | His supporters and "neutrals" hold public officials in such low regard that they call these negligibles "achievements". Disappointing, but not surprising, considering they call his admission to rape (even though proclaimed a joke by Malacañang) a fresh take on being honest. They wow at Duterte doing the bare-minimum. There's no reasoning with most of these people. Sucks that we all have to suffer for it, though.
0 | For some reason I can see the head of Donald Trump.
1 | would you say kaya ng toyota vios na slightly lowered?
0 | we complain when media only reports bad news, and then when they report something positive this happens? When someone does an oopsie, automatically every other good thing they have done is forgotten? PS. this isn’t a reflection of my political views, just my thoughts on society
1 | Unfortunately, this is still running and considered as normal "stress" lang sa Family natin. Most of my titas and titos are saying "hindi ka kasi nagsisimba", "nasa utak mo lang yan, mawawala din yan", "manage your time kasi".
0 | For the first time in a while, I cooked something for myself. For almost a week now, I've been eating just cup noodles, or bread, or eating straight from the can. Small victories, I guess.
0 | Tall poppy syndrome? lol Feign humility, say that you're lucky or smth.
0 | Don't let it bother you man, sometimes being sad is good. Made me appreciate happy times more. Oh, and she's my ex.
0 | The guy is 16 yo and he said he doesnt know of any way to ship it for less money. I'll google about DHL though!
0 | Being born and then being taught supporting your family is your responsibility
0 | But what if you're still enjoying things? Because the other person is just soo good with it...
0 | Chinese languages are more about written communication over oral one. You have to memorize at least 2-5 K characters to fully read and understand Chinese newspapers alone. Learning Chinese languages with logographic script would take entire lifetime and would be too late for any adults to learn with near-native competency with no prior exposure with said languages as a first-time learner. There's a reason why English remains and will remain the lingua franca of the Far East. let alone the entire world and that is English grammar is full of "exemptions to the rule" which makes learners able to twist the language for their own suits but not necessarily hampering mutual intelligibility with native Anglophones..
1 | Oo nga eh. Bat kelangan pa hard copies eh nasa phone mo na nga. Papakita mo naman id mo. Zzz. Useless yung paperless.
0 | Morning Wake up Browse phone Take a shower Eat breakfast Quick chores Commute 1 Commute 2 Commute 3 Work/School Afternoon Lunch Work/School Commute 1 Commute 2 Commute 3 Evening Get a nap Eat dinner Chores Watch TV/Entertainments (or Do schoolworks) Wash face Sleep
0 | I actually can't remember studying dinosaurs during my high school times. However, I do remember studying the prehistory of humans in the Philippines, which is understandable because we have human fossils in the Philippines but AFAIK we have not discovered dinosaur fossils in our islands. Archeology in general is lacking in the Philippines.
0 | Revilla: acquitted with a fine. Napoles: guilty and goes to jail. Haven’t we been expecting this? We all knew this is how it would go down, especially after Estrada managed to walk.
1 | Okay yan! Tsaka yung I Miss You - Blink 180, fam.
0 | how does dance propagates toxic shame culture and obedience? I mean dance as a PE is shitty but I don't see it. In our dance class, no one laughed at anyone, we laugh at our mistakes but we kinda enjoyed it i guess?
0 | A lot of immigrants become super proud of where they come from - especially when they go to the US/Canada or Europe - because they think it makes them cultured. "Unlike Mr. Homegrown, I have *seen* the world."
0 | I've traveled to Manila 4 times since 2015 and each time I had TSA approved locks on my hard shell luggage. Never had any problems. If someone is going to look through luggage for valuables, they will look for the easiest targets, those without any kind of lock. Better safe than sorry. Don't give them an easy target.
0 | A one-time cleanup is more of a photo op than a real solution though.Thank you! We're looking into this.
ellarabi
commented
4 years ago Hi, please re-download the entire dataset -- it contains a much cleaner monolingual part (the other sub-corpora are untouched). About 6% of the data are still code-switched (per manual inspection), but these are hard to get rid of via an automatic scan.
Please don't hesitate to contact us if you face any further issues with the data.
ellarabi
commented
4 years ago Please let me know if the data looks good to you, and I can close the issue.
stevemarin
commented
4 years ago Sorry, I didn't realize you were waiting for feedback. Re-sampling the dataset now looks much better. Thank you!
ellarabi
commented
4 years ago Thanks! Closing the issue, pls feel free to get in touch if you face any further issues.
Thank you very much for this dataset! While exploring the dataset I noticed quite a few of the entires in the monolingual dataset actually have code-switching present. For example:
According to your paper, these should be monolingual:
We also compiled an additional dataset of English monolingual posts from the same country specific subreddits as our code-switched corpus. Here, as above, we excluded posts with weblinks and posts shorter than five words. This dataset serves as an appropriate comparison set for the code-switching corpus, and is used for some analyses in Section 3.Using the FastText language detection model, the last example provides the following output:
Do you have any idea why this happened? Is this an issue with the Polyglot model, or am I misunderstanding how you define code switching?
Thanks!