Dutch data-only phone numbers are not recognized as valid

ActiveB1t commented 5 years ago

[x] I have searched open and closed issues for duplicates
[x] I am submitting a bug report for existing functionality that does not work as intended
[x] I have read https://github.com/signalapp/Signal-Android/wiki/Submitting-useful-bug-reports
[x] This isn't a feature request or a discussion topic

Bug description

Registering a standard Dutch data-only phone number is not recognized as valid. These numbers are able to receive and send SMS.

Steps to reproduce

Install Signal, or unregister old phone number
Enable Signal messages and calls option
Choose Netherlands as country (+31)
Enter data-only phone number, starting with 97 followed by 9 digits

Actual result: An error message appears, indicating the number is invalid. Expected result: The number should be recognized as valid and the next step in the registration should start, presumable receiving a verification SMS.

Screenshots

Screenshot_20190831-215648

Device info

Device: Google Pixel 2 XL Android version: 9 Signal version: 4.46.2

Link to debug log

https://debuglogs.org/c5781749fc57563f412ab5d9448eb8dd1fe55744a04d0455f9767b49d932c344

ActiveB1t commented 5 years ago

As I've figured out this issue originates in one of the Android libraries, I have submitted a bug report with them and it's been passed on to the development team: https://issuetracker.google.com/issues/140508060

alan-signal commented 5 years ago

Thanks for submitting the bug to google, but let's keep this open as our reminder for us to monitor that issue and update the version of the google library once fixed.

ActiveB1t commented 5 years ago

Will do, and thank you for keeping an eye on it.

PieterGit commented 5 years ago

Same problem for me (Wileyfox Swift 2X, Android 9 (20190801), Signal 4.49.18). I tried to use a data only SIM from my provider to register with Signal. The same SIM works fine with Whatsapp.

Signal says invalid number ("Ongeldig nummer" and "Het nummer dat je hebt ingevoerd (+31970 XXXX XXXX) is ongeldig"). I tracked it down to the Error string RegistrationActivity_the_number_you_specified_s_is_invalid which seems to be used at https://github.com/signalapp/Signal-Android/blob/0ffa10eaea334e89697d0d19f043cc26a35834ac/src/org/thoughtcrime/securesms/registration/fragments/EnterPhoneNumberFragment.java#L140-L145

@ActiveB1t I added more info on the Google bugreport. I hope Google will add 12 digit support for +31 970 numbers in isPossibleNumber()

@alan-signal Related question on the Signal policy. Does Signal support purely Machine to Machine (M2M), or only: Machine to Human (M2H), Human to Machine (H2M) and Human to Human (H2H).

4d722e48 commented 4 years ago

People from Ivory coast are also affected by this bug. In this country phone number are structured as follow: +225 XX XX XX XX It appears that numbers with a zero just after the country code (ex. +225 06 XX XX XX) are considered invalid, while other numbers (ex. +225 87 XX XX XX) can be registered without issue.

4d722e48 commented 4 years ago

It is a strange situation, because the process goes through when registering the number (ex. +225 06 XX XX XX) with another app (viber, whatsapp) ... Only signal messenger have this issue. I am wondering whether I should open a new issue for this ...

ActiveB1t commented 4 years ago

This is the same issue. Signal seems to be the only app of those that uses Android's built-in number validation. You could add to the existing issue on Google's issue tracker here: https://issuetracker.google.com/issues/140508060

PieterGit commented 4 years ago

Link to similar Signal issue on iOS: https://github.com/signalapp/Signal-iOS/issues/4442

@alan-signal or @ActiveB1t can you explain whether Signal uses the Android built in validation, or that the https://github.com/google/libphonenumber/blob/master/resources/PhoneNumberMetadata.xml metadata is used for Signal under Android. I don't understand what needs to be done to get this issue fixed for both Android and iOS.

I attempted to write a PR for upstream, see https://github.com/google/libphonenumber/pull/2499 . I am unsure if that will (after releases) fixes this problem.

panjerossi commented 3 years ago

I was going to add to this issue that the same problem exists on iOS, but see that Pieter already mentioned that.

Thinking ouside of the box, what if Signal would do 2-step approach:

A basic check (number starts with a + and only contains numbers) that then leads to a showstopper if FALSE.
If the basic check passes, do a more extensive check (the current check), not leading to a showstopper if FALSE, but gives a warning, something like "This does not seem a valid phone number for this country, are you sure you want to proceed?" And then allow the user select "Yes, proceed" or "No, Change number".

Would solve the problem and only downside is that the SMS message broker could get an invalid phone number in.

Since Whatsapp, also performing SMS validation to link an account to a phone, works fine on both Android and iOS, it should technically be possible to get this working.

jkr-wrk commented 3 years ago

I really want to switch from Whatsapp to Signal, but I can't get my family members to switch because they have a 0970 number. Is it possible to build in a second check for number validation that checks if the number is valid even when libphonenumber says it is not?

ActiveB1t commented 3 years ago

As they seem unwilling to fix the issue in Android's (in)validator, and are recommending clients to implement the workarounds they fail to deliver (see their comment here), are there any chances of a workaround being implemented in Signal? It would be nice to use Signal with the number that goes with the phone.

PieterDeBruijn commented 3 years ago

Anyone got a solution here? It's seems like an extremely easy fix (add one digit tot the allowed Dutch number length). There are lot's of people who are affected by this bug.

panjerossi commented 3 years ago

Issue is not just in Android, but also in iOS.

From Github FAQ on libphonenumber:

"Conclusion: Unfortunately we will not be able to commit for any deadline to support M2M numbers. We recommend users to implement workarounds in their client code itself."

The technical solution is not the issue; the workaround is easy enough. Is there any way to get the workaround implemented in Signal (like it is in e.g. Whatsapp)?

PieterDeBruijn commented 3 years ago

Still not fixed? Can someone please assist? This makes for seriously many new possible users.

stale[bot] commented 2 years ago

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

PieterDeBruijn commented 2 years ago

It still does not work. So all Dutch data simcards cannot use Signal currently.

ActiveB1t commented 2 years ago

I concur it is still relevant. A messaging app should implement a workaround if it enables more people to use their app instead of an alternative.

DjoeC commented 2 years ago

Agree, and it appears to be broader than just Signal and has been for a long time according to the Dutch regulatory organisation ACM

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

DjoeC commented 2 years ago

Issue still not resolved, remains important as more and more people use dataonly subscriptions. Please allow all valid numbers, not just voicenumbers.

Sgn-32 commented 2 years ago

Post from Apr 12, 2022 11:49PM https://issuetracker.google.com/issues/74493346

Subj: “Rollout of supporting NL M2M numbers as mobile in one of upcoming releases.

Hi,

TL;DR Netherlands 097 "M2M" range will be supported by libphonenumber as category MOBILE.

@Background / Objective:
We are going to support NL M2M (Machine to Machine) numbers (11 digit 97X numbers) as mobile numbers based on recommendation from Netherland's numbering authority. Evidence: https://www.acm.nl/en/publications/information-about-dutch-097-numbers-non-dutch-providers.
In NL, these numbers can be SMS provisioned end points with voice capabilities, though M2M numbers are generally for communication between machines.
M2M is marked as one usage for them, but they are also marked as H2H (human-to-human), and we are receiving a lot of complaints from users because of the fact they are not recognised as valid.
@Impact on API results:
After the change, PhoneNumberUtil.isPossible() API will consider any 11 digit number as a possible phone number in NL.
PhoneNumberUtil.isValid() API does more strict validation than the above one. This returns true only when the input 11 digit number has the right prefix, i.e 97.
@Possible breakages if you already have implemented workaround for whitelisting these numbers on your own:
This feature request has been long pending because of ambiguity around M2M standardisation. In the meantime, we encouraged clients to have their own workarounds to whitelist these numbers.
If you have done so, we recommend you to watch our release notifications (in email) and release notes on Github. If they include NL M2M range support, please fix your client code against that release version.
@What changed since 2016 and why is this (not) a slippery slope In 2016, there was a lack of official guidelines about these numbers and the majority of what we knew came from user reports. Since then, official authority has explicitly stated that this range “should be made accessible, just like other, regular number series from the Netherlands” and that “you can set up a voice and SMS connection towards prefix +31-97 in the same way as you have done already with the +31-6 series.[...] you should enable your systems for voice telephony for the numbers in the +31-97 series”. This means, however, that there might be cases where the library would categorise a number as a valid mobile number, but in reality, the particular number is used as pure M2M, is not SMS or voice-enabled. There is not much we can do from our side about this, since we always follow official guidelines.

So clients should be aware that there is a possibility of false positives in the NL MOBILE category. The library will continue to not support M2M numbers in general.

Sgn-32 commented 2 years ago

People from Ivory coast are also affected by this bug. In this country phone number are structured as follow: +225 XX XX XX XX It appears that numbers with a zero just after the country code (ex. +225 06 XX XX XX) are considered invalid, while other numbers (ex. +225 87 XX XX XX) can be registered without issue.

@4d722e48

From wikipedia: https://en.wikipedia.org/wiki/Telephone_numbers_in_Ivory_Coast

Ivory Coast has changed the phone numbers from 8 to 10 digits on 31 January 2021. A Phone number like +225 XX XX XX XX XX is working on Signal Android.

Sgn-32 commented 2 years ago

I tested it with an additional if-statement:

    if (COUNTRY_CODE_NL.equals(countryCode) && Pattern.matches("^\\+3197[0-9]{9}$", e164Number)) {
      Log.w(TAG, "Passed NL +3197 number check");
      return true;
    }

Screenshot_1650736183

But the server has also a phone number check, which then fails when verification code is requested.

org.thoughtcrime.securesms W/EnterPhoneNumberFragmen: Impossible number
    org.whispersystems.signalservice.api.push.exceptions.ImpossiblePhoneNumberException: StatusCode: 400
        at org.whispersystems.signalservice.internal.push.PushServiceSocket$VerificationCodeResponseHandler.handle(PushServiceSocket.java:2505)
        at org.whispersystems.signalservice.internal.push.PushServiceSocket.makeServiceRequest(PushServiceSocket.java:1668)
        at org.whispersystems.signalservice.internal.push.PushServiceSocket.makeServiceRequest(PushServiceSocket.java:1652)
        at org.whispersystems.signalservice.internal.push.PushServiceSocket.makeServiceBodyRequest(PushServiceSocket.java:1641)
        at org.whispersystems.signalservice.internal.push.PushServiceSocket.makeServiceRequest(PushServiceSocket.java:1577)
        at org.whispersystems.signalservice.internal.push.PushServiceSocket.makeServiceRequest(PushServiceSocket.java:1565)
        at org.whispersystems.signalservice.internal.push.PushServiceSocket.requestSmsVerificationCode(PushServiceSocket.java:315)
        at org.whispersystems.signalservice.api.SignalServiceAccountManager.requestSmsVerificationCode(SignalServiceAccountManager.java:240)
        at org.thoughtcrime.securesms.registration.VerifyAccountRepository$requestVerificationCode$1.call(VerifyAccountRepository.kt:49)
        at org.thoughtcrime.securesms.registration.VerifyAccountRepository$requestVerificationCode$1.call(VerifyAccountRepository.kt:31)
        at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43)
        at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
        at io.reactivex.rxjava3.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
        at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:38)
        at io.reactivex.rxjava3.internal.schedulers.ScheduledDirectTask.call(ScheduledDirectTask.java:25)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:920)

https://github.com/signalapp/Signal-Android/blob/efe7b3099fb67f72334d0de08f60bceb2c6f51f1/app/src/main/java/org/thoughtcrime/securesms/registration/fragments/EnterPhoneNumberFragment.java#L238-L242

Sgn-32 commented 2 years ago

libphonenumber 8.12.48 now supports dutch +31970 numbers

  <!-- As per this government.nl's doc some sub ranges of existing 06[1-58] mobile numbers are
       used for M2M purposes (that use mobile networks). But there is no information about these
       sub ranges. The other way is also true, 12 digit 097 M2M numbers are used for mobile
       purposes. More information in https://www.acm.nl/en/publications/information-about-dutch-097-numbers-non-dutch-providers
       Added support just for subrange 970X as these only mentioned as assigned/live. -->

Testing is possible with the java demo:

https://libphonenumber.appspot.com/

Sgn-32 commented 2 years ago

https://github.com/signalapp/Signal-Android/commit/12d1254d4e9b1e9b37bc58243608ed6ae9ca10cd in 5.38.0 beta

Sgn-32 commented 2 years ago

Signal Android and Signal Server are updated to libphonenumber 8.12.48

So registration with +31 97X XXX XXXXX is possible.

Screenshot_1652370426

signalapp / Signal-Android